No Deploy Fridays at CrowdStrike and the Mendix Marketplace update lifecycle

If you are reading this title and think CrowdStrike, don’t you mean that old-school shooter game everyone was playing in the early zeroes? You might want to open a random news website and start reading. Or let me at least give you a small digest. 

Last Friday (19^th of July) cybersecurity company CrowdStrike rolled out an update for one of their software packages called Falcon Sensor. Software that is used by governments, large enterprises, hospitals, and airports to secure their machines. These types of organizations are usually organizations with IT departments that manage all the hardware for their employees. Well, as our newest podcast is called No Deploy Fridays for a reason, this roll-out went terribly wrong and caused all computers that used this software to throw BSOD (Blue Screen of Death) to its end-users and causing a boot-loop, meaning if you worked at one of the organizations mentioned above, your weekend started early or your holiday started with a delay because complete airports were crippled by this malfunction. 

There are already plenty of news articles and blog posts out there about what happened, and I will be the last person with any knowledge on this subject to say something meaningful that hasn’t been said already. 

No Deploy Fridays

But it made me think of an earlier conversation with a Mendix customer we were helping with the platform a day earlier. We were going over how they could best manage their app lifecycle, and specifically those of marketplace modules. This conversation ended up in our suggested strategy for how and when to use certain Marketplace modules and most importantly when to update them. So, in a sense, this might be related a little bit to the global outage of critical computer systems, but I’ll be honest there’s a scent of click-bait in this title.  

In our first-ever episode of No Deploy Fridays, Paul Ketelaars and I interviewed Maurits Elzinga on the topic of the new Mendix Extensibility Framework and we asked a question on governance and how you could prevent Bitcoin miners from entering Studio Pro (if you haven’t listened yet you can here:

And the answer was apparent. You yourself are responsible for what you’re adding to your model and Studio Pro. There’s of course a review process in place for what’s entering the Mendix Marketplace, but there’s more to it than just checking source code and adhering to guidelines. When we were going over the lifecycle of components in the Marketplace, we shared a few tips that might help you as well. 

The Marketplace lifecycle components

The first thing we shared was a simple one. Look at the support label. There are currently five types of support labels:  

1. Platform – these modules are maintained by Mendix, adhere to SLA’s, and follow the update lifecycle path of Studio Pro. Meaning, it should be compatible with the latest LTS. 

2. Deprecated – as the name might suggest, these modules will be taken offline, and you should look for a replacement. There’s usually a suggestion in the marketplace description for what module you can use best. 

3. Community – is probably the most used label you can find. Anything uploaded by the “Community” e.g. content from Mendix Partners or teams within Mendix that share content without a fixed support plan. 

4. Partner – This label is relatively new and sits in between Platform and Community. It is content developed and shared by a Mendix Partner but with the support of an SLA. These listings could involve licenses as well. The Bizzomate Ultimate Scheduler is a great example of this.  

5. Siemens – Last but not least, content released by Siemens and supported by Siemens. Same as Partner, but with a clear notion this is from Mendix their parent company Siemens.  

Now with Platform, Partner, and Siemens you can assume all is good when you download/purchase content from the marketplace. Deprecated modules speak for themselves, if you keep on using them, you’re going to make life difficult for yourself. But with community-supported content we always ask ourselves the following questions before we download anything from the Marketplace: 

1. Is the module frequently updated? Because if a module had its last release two years ago you could ask what would happen if a security issue occurred in the module and if it would be fixed. 

2. Does the developer still work for the company that released the module? You could be surprised but there are modules in the Marketplace released by Company X and Developer Y, but Developer Y now works for Company Z which does not own the source code of the module. This could pose issues as the knowledge of this module might leave the company as well.  

3. How are the reviews? There might be 20 five-star reviews on a marketplace listing, but as some Mendix developers might recollect, there was a Swag store (RIP) back in the day and you could earn points for your general Mendix level but also for buying amazing Mendix socks from that store, resulting in a lot of “Great module” reviews by the same person repeatedly on all different modules.  

Trusting the content you download

There might still be a risk in downloading content from the Marketplace even if you did your due diligence as best as possible. A few years ago, when the world was confronted with another global IT circus revolving around the Log4J vulnerability, Mendix wasn’t spared as well. Quite quickly all platform-supported modules were updated and got a fix for this vulnerability. You might also remember emails from Mendix support listing your affected apps that contained the Log4J JAR file that included this vulnerability.  

It meant that all community-supported content had to be updated as well and this meant also going over modules that might not have received an update for years. There were applications in which we had to remove modules because there weren’t any updates available, or we had to build fixes for them ourselves. Reinstating that to use Marketplace modules you should always have a good strategy in place to make sure you’re not just downloading the next security vulnerability or bug into your Mendix application. 

Making sure you trust the content you download is one of the most important steps in keeping your Mendix application safe. However, once updates are piling up, there’s another problem you need to consider. Have you ever checked the userlib folder in your Mendix project? Or the widgets folder? If you haven’t, you might be surprised by the number of .jar files in there. JAR files contain the dependencies used by Java actions in the Marketplace modules you download. Unfortunately, Mendix doesn’t (yet) automatically manage this folder for you. So, if you download a new version of the Community Commons module, and this new update introduces an update of the commons-logging jar file, it won’t clean up the old one and this old jar file keeps living in your Mendix project, which could pose security risks as over time, these jar files might contain vulnerabilities. Therefore a good practice is to, after every marketplace update, check your userlib folder and widgets folder for old versions of JAR and widget files and to clean them up. 

Being proactive

Anything you’ve read of course means nothing if there will be a next Log4J vulnerability or any other worldwide critical bug, to be honest. But making sure you know what you’re pulling into your Mendix applications and thinking twice about how this is maintained might save you an awful lot of time and grey hairs the next time you receive an email with all the Mendix apps affected by a given issue. 

Instead of waiting for Mendix to send you a support mail with the next vulnerability, you can proactively check for these security risks yourself by implementing automatic vulnerability scans on your Mendix projects. Using tools like Snyk, you can monitor the dependencies used by Java and JavaScript actions and used by widgets. Once a vulnerability is reported by Snyk you will automatically get notified about this, including a mitigation plan, if this is available. 

The list keeps on going. Things like always staying within the list of supported Mendix versions, to prevent upgrades from failing. Don’t just deploy on the latest version of Mendix, but keep production deployed on an LTS (Long Term Support) version so you know that if things go south with the Mendix Runtime, you have Mendix support to rely on. 

To summarize, there’s a lot you can do yourself to protect your apps from becoming a security risk, where the most important part is to take responsibility yourself, think twice before you download a module from the Marketplace and proactively keep an (automatic) eye out for any risk. And hopefully, at CrowdStrike they will now think twice when they deploy on a Friday.