“Knowing Your Software Inside Out is Key”

There are many lessons to be learned from last year’s Log4j vulnerability. One of the more important (and maybe underappreciated) ones is that there are no easy fixes for complex challenges: Only a couple of days after the official patch had been published, experts all over the world documented multiple scenarios in which Remote Code Executions via Log4j were still viable even after the fix. One of the first cases was the vulnerability CVE-2021-45046 [1], discovered by iC Consult’s Kai Mindermann. Kai kindly sat down with me for a quick chat.

Heiko Klarl (hk): When the Log4j team released the official patch in December, IT practitioners all over the world breathed a collective sigh of relief, patched their systems, and returned to normal. When did you hear about the vulnerability, and what kept you digging deeper?

Kai Mindermann (km): Thanks to our close relationship with several leading vendors, our team had been among the first ones to hear about the vulnerability. We had discussed the implications in-depth, first with our team and then with our customers – and we realized right away that Log4j’s damage potential was huge. As a Project Lead, I am currently coordinating a major Managed IAM (Identity & Access Management) Services implementation for one of our biggest customers.

Since I knew that Log4j was an integral part of their solution, I wanted to make sure that the vulnerability had been fully closed. By mid-December, several reliable sources had suggested different plausible fixes for the issue, and to be on the safe side, we decided to implement all of these, or at least as many as we could. We started with the recommended update to version 2.15. But once we looked into the updated system up close, we quickly discovered that our highly customized environment – which was using Log4j with Thread Context Maps – was quite different from the default configuration. Basically, our way to parse variables into the log format was completely different, and we quickly noticed that even after the fix had disabled message lookups, those could still occur via Context Lookup. This was very worrying, a first indicator that the update was not enough for our specific scenario.

hk: What were your next steps?

km: The first step was to adapt our Custom Log Format to make sure that the fix worked as intended. Then, we focused on the communications: Our primary concern was to warn our colleagues that one of the proposed fixes might not fix the vulnerability. While I spread the word, my colleague Tim Beierlorzer reached out to the partner who had also shared the original mitigation suggestions with us.

Then, with the loophole closed, our customers safe and our colleagues informed, I could look deeper into what we had uncovered. I ran several more tests to make sure that my suspicion was correct and alerted the Log4j security mailing list about the new vulnerability. I didn’t get any reply right away. As I felt this was urgent for others to know, I sent another email – which also included my final validation and a Proof of Concept – and set a twelve-hour deadline. Still, I did not hear from them in time, so I contacted several trusted peers and asked them to validate my findings which they quickly confirmed. Around that time the Log4j developers presented their changes to the official mitigation strategies of the original CVE-2021-44228 and published the newly discovered CVE-2021-45046. Then, the community got to work.

hk: How long did the entire process take?

km: Just a little over a day. But a day can seem very long under these circumstances.

hk: And once you had the confirmation…

km: Once we had the confirmation, we warned all colleagues at iC Consult that they must not rely on the official patch recommendations in their projects, but should rather migrate to the latest version 2.16, or simply deactivate the Thread Context Maps for the time being. And we also stayed in close contact with our customers and multiple other communities and asked them to spread the word and make sure everyone knew about the issue.

hk: And that did the trick?

km: As far as I know, it did. I do not think any of our customers or partners ran into any problems. And as far as securing our own applications is concerned, one of the advantages of running the flexible and code-based platform like Service Layers is that we understand all apps inside out and can fully control any aspect of the software. Once our teams understood what to look for, they knew exactly which systems were affected and how to fix them.

hk: That is good to hear. But still, the whole scenario remains very scary: If we cannot trust a vendor patch to protect us, what do we have left?

km: I think this is another major takeaway from Log4j: Today’s software is always a collaborative effort, and most applications are so complex that not even their developers know all their intricacies anymore. Also, the high degree of customization and complexity makes it virtually impossible for a vendor to test all possible scenarios and configurations. Both of this are totally acceptable, in my opinion – but it is something we need to take into account when speaking about application security.

hk: But you can stick with the default configuration and stay on the safe side, right?

km: It depends – on the one hand, you can mitigate some of the risk by using default systems. On the other hand, good software should always strive to provide a great solution, and that will usually require some degree of customization, especially when looking at Libraries which need to cover a lot of different scenarios. Also, many customer requests revolve around adding new functionalities or covering specific new use cases, and that also often means deviating from the default setup. So, in the end, software is bound to become more complex in the future, and that will sometimes cause security-related issues. One possible solution – especially regarding Open-Source projects – could be to separate new functionalities from the original distribution. But that approach comes with another set of problems – like a growing number of dependencies and APIs –, so I think that we really need to brace ourselves for increasingly complex environments.

hk: That means another Log4j-type vulnerability might wait right around the corner. How can we prepare for that? Which strategy lessons did Log4j teach us?

km: In general, IT teams did a very good job when the crisis struck. I was really surprised how tightly the communities collaborated, how quickly know-how was transferred, and how smoothly the fixes were released. That was impressive, and I think this was in large part due to the commitment and devotion of the Cyber Sec communities. I can only speak about my team, but they put in a ton of extra hours without any hesitation at all – first to analyze, identify and mitigate the vulnerability, then to inform and support our customers. Collaboration, communication, and transparency were key to the solution. In addition – credit where credit is due –, the Open-Source communities did a lot of heavy lifting, too. I am not sure if a vendor’s proprietary software would have been fixed so reliably. That said, with Open Source being such an important part of today’s applications, the industry really needs to put a much stronger focus on securing and validating Open-Source Software – the responsibility needs to be distributed across many more shoulders.

hk: Let me wrap up with a more personal question: Considering that thousands of developers and security researchers were looking very closely into Log4j over the past months: Why do you think you have been the one to discover the vulnerability? What did it take? Instincts? Persistence? Or pure luck?

km: Probably all three, in equal measure. It is a detective’s work: One must be curious, approach the problem creatively, and keep poking the software from many different angles until the riddle is finally solved.

hk: You make it sound easy. But to solve this riddle, you had to dive deep into the code. Isn’t that a bit unusual for a project lead?

km: No, not at all. As a Project Manager, Consultant or Developer at iC Consult, you know a thing or two about modern application development. If you want to really understand a customer’s challenges and requirements, a deep understanding of their software is a useful skill.

hk: Thank you so much – not only for these valuable insights, but primarily for the exceptional work you and our teams have delivered in keeping our customers secure.

References

[1] https://logging.apache.org/log4j/2.x/security.html – CVE-2021-45046: Apache Log4j2 Thread Context Lookup Pattern vulnerable to remote code execution in certain non-default configurations (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046)