Enter the commoditized IT, pushing the boundaries

Commoditization of services might sound like a new trend. After all, all those Ubers and Airbnbs seem like fresh ideas, appearing just recently. We still call them “market disruptors”, right? But probably only those working with IT every day realize that the same trend, with aim for the smallest item performing an unique function, has been present in the IT industry for decades. In this article, I will present the trend as I see it and share my observations. For the casual reader – I promise to keep it as user friendly as possible!

With a certain level of simplification, I believe I can make a statement commoditization in IT started when the “distributed environment”, as the IBM people call it, took off with the approach of creating simple, stand alone computing platforms we refer to now as servers. The attempt to offer services installed on multiple servers, dispersed globally for optimal customer experience and resilience is still popular even with the obvious faults of this approach. For example, servers had a set of resources at their disposal, as the vendor designed them to be applicable for majority of use cases. That usually doesn’t match with what the hosted application really needed. These servers were not as resilient as the mainframe, but at a reasonable price, they allow much faster concept-to-software-service process. These inefficiencies naturally lead to costs that normally wouldn’t have to be implied, but also to indirect outcomes such as developers, especially in the high performance computing sector, having to perform low level optimizations of their software.

The second level of infrastructure commoditization I call the concept of defining key resource areas commonly used by software and creating platforms which allow administrators looking after resource pools. Typically, this is referred to as server virtualization. Imagine your developers work on two projects where the product in each case is an application, but with an entirely different behavior. An application doesn’t store much data, but performs difficult computations – no problem, the admin can spawn (yes, spawn!) a new server with some memory but multiple processors (as long as the app can make use of them). The other application does little math but needs to load huge objects into memory? The admin could now use the memory resources saved on the first project and allocate them here. Running out of resources in the pools? Just insert (hot-swap) another blade of CPUs, RAM or expand the SAN for data storage.

Around the same time, the CPU itself has undergone even further commoditization, we could call it level 3. Multiple solutions have been implemented on processors to improve their efficiency without increasing the clock speed or when further miniaturization of the manufacturing process was not yet achievable. One of the key optimization areas was based on the observation that a lot of potential is lost during the processor cycle itself. As a result, various instructions have been implemented, like SMT for physical cores of a CPU, HT for logical cores, but also VT instructions for the actual virtualization. As a result, we get logical cores in physical cores forming a modern CPU.

Level 4 is a more recent tale, when the definition of SDNs, or “software defined networks” appeared. In simple words, the virtualized stack of standard commodities could be expanded to cover the network as well, which in the internet era is a commodity in itself. The idea is to present a complete set of infrastructure items to meet the development design, which naturally speeds up the whole infrastructural design phase of a project and most importantly, offers a great deal of simplicity to everyone involved in the development. Deployment of new applications of services has never been this easy.

With software defined networking and pools of virtual resources at hand, it is not hard to notice this approach still borrows from the good old “application hosted on a server” concept from what I called “level 1” era. This means each virtual server comes with a lot of overhead, including the operating system and its “wealth” of features. An OS is usually prepared with the idea to just work for as many setups as possible, which then by definition makes it “not the most optimal piece”. Level 5 of commoditization comes to the rescue – containers, among them the most popular container technology – Docker. A single server can run multiple containers, which on the outside look like virtual machines themselves, however they contain only the bare minimum of the OS that the applications hosted in them need to operate. Brilliant, isn’t it? Add to that software to operate multiple containers such as Kubernetes.

So what’s next? Commoditized infrastructure pieces, commoditized operating system allow granular choice of elements to operate applications or services. It doesn’t look like much more can be achieved here. Who can further push the boundaries? The answer is based on the fact that, just like the OS, almost all concepts in popular technology are presented in a way to allow 95% of the use cases to succeed. But what happens, when you’re in that remaining 5%? And who usually ends up there? For the last decade, in my opinion, two areas fall in the 5%: high frequency trading and high performance gaming. Add to them extreme data center owners (aka hyperscale), such as Google or Facebook and you’ll notice they all already have the solution. Software will only be software – it gets loaded into standard memory, operations executed using a standard core. But if the same operations applied to a virtual container happen million times per day on a standard piece of commodity, why not move them to hardware?

FPGA, or field programmable gateway arrays are becoming popular, because they allow just that. The software developer’s work doesn’t have to end as soon as the code is sent to the compiler. The compiler allows the code to be loaded and executed by a typical CPU, which by design is universal and can do a lot of things. Most software however perform a set of standard operations and then something unique to their nature, which gets repeated millions of times. FPGAs allow commoditization of that unique, repeatable activity, which when compiled onto the programmable board, can give it a performance boost of 10-100x at only 20x power increase. There is an immense cost reduction, instead of simply scaling the pool of CPU/MEM resources.

Does it mean application programming has to again become very low-level, make a huge turn around from the era of rapid development? Not at all, there are tools available that make it very easy, for example by utilizing Verilog. Certain standard blocks (DSPs) are pre-installed on the boards, to allow the developers to actually implement only the higher level logic. At the same time, programmable boards are available at reasonable prices and they can be re-programmed, unlike the super-expensive boards produced on demand for HFT trading companies.

FPGAs are the next big thing – if the above combination of breakthroughs did not convince you, take into consideration the battle to purchase Lattice, a vendor of FPGAs only, for $1.3 billion. Consider the rise of Xilinx, specializing in programmable SoCs. Last but not least, Intel acquired Altera about a year ago with the goal to join precisely this promising business.

So what the CIO/CTO can do? While it might not be very easy to find Verilog/VHDL specialists who also are capable developers understanding time-to-market and quality programming concepts, it was a matter of time until other vendors try to fill in the gap. Amazon is already offering EC2 instances with programmable hardware. OVH also offers a similar solution called RunAbove, which at the time of writing this article got completely sold out. Last but not least, there’s also Reconfigure.io, offering a Google Go language compiler optimizing the code developers wrote to be directly installed onto programmable boards.

What’s next? Maybe containers hosting micro services could be moved to FPGAs?

3 questions to…

Here’s something much easier to read hopefully, compared to the previous, quite technical posts. MoneyGram International runs a “3 questions to…” series in the corporate newsletter with the aim to bring the team closer and shed some light on the profiles of senior leadership in the company. My turn to answer 3 questions came last week and I’m glad, because that’s always an opportunity to practice the non-technical writing. I didn’t expect much out of it, but the feedback was exceptionally positive. Well, positive enough to convince me it might be worth sharing with LinkedIn. What do you think?

You’re managing multiple IT Teams at MGI. What skills do you find exceptionally useful in your role?

Empathy, the queen of all skills is probably worth mentioning on the top spot. This is especially important if your teams work from all around the globe and come from a variety of cultures. Leading such teams is often a challenge, even in IT, where information is crisp and precise. Same message is worth conveying in a variety of ways, staying considerate of how others might look at challenges or upcoming projects. Likewise, sometimes substantially different encouragement is needed in order to gain valuable input and uncover true potential of individuals. Being considerate about those differences and yet not losing track of the technical bits and the end goal is a great challenge to have. I believe that empathy is the key to victory in this aspect.

What are the biggest challenges you face at MoneyGram and how do you address them?

I’m sure everyone mentioned time zone difference between Europe and the US, so I’ll skip that and mention something from technology perspective. As such, while we deliver interesting solutions to the market, I believe we can still do better from the perspective of internal and customer facing technology. While staying abreast is an area dedicated to roles outside of IT Operations, I’m putting extra efforts to bring up some ideas and gain resources to new projects.

Based on your professional experience, what career advice would you give to your colleagues at MoneyGram?

I try to give advice to my team (then it’s called “direction”;) ) all the time, so to give one piece of advice now in general is not an easy task without sounding trivial, but let me give it a try. I believe that many people underestimate the power of research. For example, every time I hear or read anything I don’t know or am not sure about, I make sure I write it down. I then have a dedicated time slot, negotiated carefully with my wife, where I follow up on each item on my research list. Sometimes reading is not enough and I end up like building a proof of concept. I never let information get past me without verification – if there’s a chance to broaden knowledge in the area I’m working in, then it is certainly worth following through. Having strong understanding of all aspects results in working less on assumptions, which gives confidence a great boost. And what’s a leader without confidence?

Could SHA-1 and Dirty COW have something in common?

SHA, or Secure Hash Algorithm is a hash function initially announced in 1995 by the United States’ NSA. It become widely popular with appliances for password hashes, certificates and a variety of software. The implementation made it to SSL (and later TLS) collections, SSH, PGP and many more. From the very moment of publication, SHA-1 was contested by cryptography researchers from all over the world, but not to much success. It took 10 years after the initial publication for the first papers to appear claiming weakening SHA-1 to some extent. The papers remained theoretical due to the cost – it would take almost $3mil of CPU power to execute the attack. It took another 10 years for a group of security researchers to further weaken SHA-1 and actually demonstrate an attack by moving the computation to 64 Nvidia GPUs. And why am I writing this? Mainly because this is a story of great success – for a hash function – to remain unbroken for over 2 decades. Still, around 2015 various security authority boards have decided that, by Moore’s Law, within the next 2-3 years, the cost of attacking SHA-1 by brute force was going to be economically viable and generally available. Based on that SHA-1 was to be gradually removed from public use.

On a side note, here’s one very intriguing case with lots of food for thought. Due to the wide adoption of SHA-1, the ban was instated by various decision makers at different times. In 2015, the Browser Forum decided that SSL certificates using SHA-1 can no longer be issued after the 1st of January 2016. Was this enough to stop all CA’s from issuing the popular SHA-1 certificates, sometimes even hard-coded into proprietary software? Dit auditors save the world as it is widely expected by various company boards? Read the full story and brilliant analysis here.

Back to the original subject, I got a bit nostalgic for a reason, and it is not because I hadn’t had steak for a while. Dirty COW is a funny acronym for a rare race condition introduced into the Linux kernel Copy-On-Write code around 2007 (kernel 2.6.22). It is not however a new vulnerability. Linus Torvalds revealed that even though it was discovered long time ago, the race condition leading to privilege escalation was rare enough that it was actually discarded. It was concluded that computation resources required to trigger the condition are not realistic. And almost 10 years later anyone can execute a local exploit and bypass internal security countermeasures companies spend huge sums on. The exploit is easy to obtain, execute and has a very high success rate.

Two completely different cases and security risks. And a complete contrast in strategy. Or the lack of. A decision to obsolete SHA-1 well in advance of vulnerabilities, despite a decent design. And no decision for a ticking bomb. I have read many quotes by Linus on security (or actually against security). Don’t get me wrong, he still is a brilliant lead for the kernel development, but why wouldn’t the Linux Foundation add the missing strategy? Track security bugs that are closed for a reason like this one and react on time? Someone might say “that’s not their job”. If reality requires it, it has to become someone’s job.

VoIP idea: foreign number to your mobile

I realized I haven’t written anything about VoIP yet in this blog. It is a very interesting subject and a great way to deliver your employer decent “buck for bang” once you get to know how it works. In this post I will explain how to configure a public DDI (direct dial in) number to point to your mobile phone.

Tools used

  • Asterisk PBX
  • iptables
  • SIP/IAX2 client for Android/IOS (anything from the app store)

Use cases

  • Give people (friends, relatives, coworkers) a number that is local to them to cut their costs.
  • Use a foreign number to register to foreign services.
  • Show off 😉
  • EDIT: this solution had some isssues. Ironing them out is explained here.

    The idea is to get a DDI via SIP trunk through any VoIP provider. Some used to offer this service for free, I got it from a good colleague in London. Yes, I will be using a London based number in this case, but the geo-location is agnostic to this proof of concept. Here’s how it would work:

    [PSTN Cloud] –(POTS)– [VoIP Vendor] –(SIP/Internet)– [VPS/Asterisk] –(SIP/3G)– [Mobile/SIP-client]

    PSTN Cloud is the network over which regular landline phones operate. POTS is the plain old telephony system, again – the landlines. VoIP Vendors integrate POTS with the Internet and as a result I can purchase a DDI like (+44) 0207 1234 567. The VoIP Vendor will pass calls to my number by initiating a SIP session to my VPS (virtual private server) IP address, where I will have my Asterisk PBX listening on the SIP and RTP ports specified.

    I will summarize security concerns later, for now focusing on the functional side. I use the SIP protocol in this example, although IAX2 is a good alternative. For Asterisk to handle incoming connections, little configuration is needed. I need to have two peers configured in sip.conf. One for my vendor, and one for my SIP client on the mobile phone. The SIP client would. as per the standard, register every 60 seconds to ensure Asterisk has a recent enough IP address to which it could redirect incoming calls. Here’s an example (comments start with semi-colon):


    root@rzski ~ # cat /etc/asterisk/sip.conf
    ; general, security and performance settings
    [general]
    bindaddr=your.ip.address
    alwaysauthreject=yes
    allowguest=no
    videosupport=no
    srvlookup=no
    transport=udp
    tos_sip=cs3
    tos_audio=ef
    cos_sip=3
    cos_audio=5
    language=en
    tonezone=uk
    useragent=Asterisk_Gentoo
    dynamic_exclude_static=yes

    [context_for_mobile]
    contactpermit=0.0.0.0/0.0.0.0
    type=friend
    disallow=all
    allow=alaw
    allow=g726
    allow=ilbc
    allow=gsm
    canreinvite=no
    qualify=yes
    nat=no
    dtmfmode=rfc2833
    defaultuser=your_username_here
    secret=your_password_here
    host=dynamic
    context=nonexistent ; in case someone gets up to here, they will not be able to make any costly calls.

    [context_for_VoIP_vendor]
    type=friend
    disallow=all
    allow=alaw
    allow=g726
    allow=ilbc
    allow=gsm
    host=their_ip_address
    canreinvite=no
    qualify=no
    nat=no
    dtmfmode=rfc2833
    context=INCOMING

    Then, naturally Asterisk needs to know how to handle the incoming connections. This is achieved in extensions.conf.


    root@rzski ~ # cat /etc/asterisk/extensions.conf
    [INCOMING]
    exten => 442071234567,1,Dial(SIP/context_for_mobile,,)
    exten => 442071234567,n,Hangup()

    There are hundreds of other settings that Asterisk can use, it’s power comes from the excellent customization options and flexibility. I recommend reading the well commented configuration file samples bundled with the source code, as well as visiting the online documentation on Digium’s wiki pages.

    Finally, I’d need to configure the SIP client on the mobile. All it takes is to specify the IP:Port, username:password of the VPS and job done.

    Challenge: Security

    I mentioned I will use iptables. Asterisk is great at many things, but it only has basic countermeasures for cyber threats. It is optimal to deal with the security issues as much as possible on the network level and leave the session security and functionality layer to Asterisk.

    I try to keep my iptables neat – default policy is set to DROP and only a few rules to enable chosen traffic. While for the vendor-VPS communication it is fairly simple – I can white-list connections based on the source IP address (the vendor’s) and the destination port/protocol (to accept SIP and RTP traffic only from them), the other half becomes a bit more problematic. Over 3G or LTE, my mobile can pick a lot of various IP addresses.

    I could narrow the area down using GeoIP module, which would allow me to connect from a single country only, but that’s still not ideal. I’m open for a whole country and, what if I travel?

    Another option is to simply tighten the password and use fail2ban. While fail2ban works great in many areas, on a busy server having a system daemon working on 50 log in attempts per second is just not the type of workload I want on my server. Plus, they all get a few tries before future connections are dropped, polluting the log files.

    I believe security is always a compromise of performance/comfort, but should always be considered depending on the use case. In my case, I control 2 ends: the VPS and the mobile. All I need to do is ensure these two have a token they both know about and filter everything else out. Such token could be a certificate, but I’d need to spend some time coding that into the SIP client. I decided to use the string matching module for iptables to only allow traffic coming from my client based on it’s UserAgent ID. Each (well written) client tries to identify itself with a vendor string when communicating with the server and so do many SIP clients. Even better if that string can be configured – in such case you can append some non-trivial string which then you can tell iptables to allow on exclusive basis.

    To obtain or verify the string used in communication between the phone and the server, it would be required to temporarily enable SIP debug (sip set debug on) and review the messages.
    Iptables configuration:

    ### Mobile SIP and RTP access
    # SIP
    /sbin/iptables -N SIPREG
    /sbin/iptables -A INPUT -p udp --dport 5060 -j SIPREG
    /sbin/iptables -A SIPREG -m string --string 'User-Agent: whatever' --algo bm -j ACCEPT
    /sbin/iptables -A SIPREG -j DROP
    # RTP range
    /sbin/iptables -A INPUT -d $PUB_IP -p udp -m udp --dport 5500:5600 -j ACCEPT

    And in case something starts spoofing the string somehow (what’s the chance) and try to guess the password, we can throttle that:

    # SIP brute force
    /sbin/iptables -I INPUT -p udp --dport 5060 -m recent --set --name ASTERISK
    /sbin/iptables -I INPUT -p udp --dport 5060 -m recent --update --seconds 15 --hitcount 10 --name ASTERISK -j DROP

    Note the SIP user agent string check is in a separate chain to allow managing this traffic in the right order.

    One potential pitfall would be if your phone updates the client and thus re-sets the string you’re looking for. Matter of caution.

    And that should be all needed to get inbound calls directly to your mobile. Questions / comments / concerns?

    Device-as-a-Service from HP

    Hewlett Packard have announced a new type of service called DaaS, or Device-as-a-Service, based on which for a fixed regular fee, enterprises could use PCs and printers. Makes sense to me, but what part of this service is actually new? Allowing others to use property is known for centuries under different names: renting, leasing, etc. There are many legal forms and the difference is usually in the approach to ownership.

    Ownership in most cases comes with costs and responsibilities which enterprises but also end users are better off ceding to others, who specialize in the matter and are able to implement optimizations to make the business profitable. Most times the issue is with cash flow, the possibility to spread a large expense over time. A good example are enterprise class printing multi-functional devices. They are big, complex and expensive, yet every medium to large firm needs at least one to function today. What made that possible was the multiple capex and opex options of contracting. Another example are mobile phones added by vendors to call plan contracts, nothing unusual.

    So what is different now? I have read many comments about HP simply trying to hook up to the catchy “X-as-a-Service” buzz word. I think it is undeniable that whoever launches a business line based on anything they can call “as-a-Service” is making a good commercial for themselves these days. Nevertheless, isn’t there a major gap on the market for something that seems to make perfect sense? In most enterprises now, every employee is a user of the IT infrastructure and is very likely to be equipped with some end point device. These devices, multiplied by the number of employees (assuming 1:1 ratio) continue to be a major cost. And that again, multiplied by whatever is defined in the technology lifecycle management policy of the company they work in. Given the above, why aren’t most enterprises moving towards a (ok, I’ll bite…) DaaS model for PCs? Perhaps it is not a well known option and the buzz word approach has the chance to change that? If that works, HP are going to be very profitable, as well as all their partners who are going to offer this as a local service.

    Nevertheless there’s something that seems missing. I read the articles and well, they all mention enterprises. Why the limitation? Not long ago I equipped my parents with a Linux PC, ending an era of dealing with performance issues and virus infections on their home PC. While it didn’t take much time to prepare, I’ll probably need to do some upgrades once the new HD, 4K or other fancy buzzword kicks in. If I had the possibility to sign a good desktop support service for non-tech savvy relatives, I’d probably go for it (or let them do it by pointing them at the offer). A colleague recently made a good comment I could not agree with more: the world has more and more technology users and fewer and fewer of those who actually understand the technology and are able to repair it when required – similarly to car drivers and those who understand what happens under the hood. The answer is moving to shared services – an approach enterprises are already adopting. I presume, households will follow soon. I will also add this to my list of business ideas 😉

    Is there a future for 3D printing?

    I want to dedicate this post to innovation in technology and entrepreneurship. The question in the title is probably a silly one because a recent event taught me that 3D printing is actually already a 30~ year old idea. In the recent years however it managed to break into the mainstream, probably thanks commoditization and availability in the retail area. This seems to have worked as an eye opener for the wider audience, including people ready to invest into this technology – and including myself.

    Like many people, when I read about 3D printing, I tried to find additional information describing the process, innovation and use cases. Nevertheless whatever I found did not really explain all the hype powering this subject. I mean – OK, it is a new thing, allows creating, designing, but where is the big selling point? Is it a matter of having a new toy for the kids with focus on boosting their creativity? Or maybe this is all about small markets for catchy goods like these rings from Japan? For a long while I could not find anything better beyond that. What further made me think “oh well, time to move on to other things” is a post on Slashdot about a year ago, asking the readers to share their use cases and ideas. The result? Many comments simply said “nothing”. Some brought up the possibility to reverse-engineer and produce parts which are no longer in sale. That’s something, but not much. After all, the trend today is to make devices that only last a couple of years. Plus, not every material can be used in the 3D printing process.

    The actual eye opener for me came on the recent CIONET event dedicated to start-ups (kudos to Krzysztof Frydrychowicz and Bartosz Górczyński at this point for making every CIONET event so worthwhile coming to). Among the start-up presenters was one not so much of a start-up but a lively developing company called 3D Makers Zone presented by its founder, Herman van Bolhuis. This Dutch company was a start-up in the 3D printing business in Amsterdam not so long ago. Now they have a solid, diversified business and are looking to scale up. I have to admit that with the outcome of my previous on-line researches I was rather sceptic. Nevertheless after Herman’s passionate presentation I couldn’t help myself but approach him for a discussion. The goal for me: understand what he sees in the 3D printing business that I am currently missing.

    The first mistake might be obvious by now. I had focused on the retail market, for which there might be no better future than what I mentioned above. The real business is the niche that remained between contracting factories for mass production and the smaller businesses, that needed affordable and easily available production on a smaller scale. Imagine a workshop on your street where you could go and have almost anything you need designed and printed with a decent level of quality in 50-100 pieces only. If you run a repair shop or anything similar that could be a godsend.

    But Herman takes it further. With the miniaturisation of technology, he’s able to produce “smart” materials – with embedded sensors, or even something that can change its shape depending on environmental circumstances. But what really impressed me is what he called “distributed manufacturing”, meaning production of goods at their destination. Taking into consideration the amout of items that are easily produced but then their price increases drastically due to shipping – this is just brilliant! I need a full ship cargo of mugs I produced shipped to Brazil? No problem, have a printing shop set up across the Pond and all that becomes easy and without the risks or insurance costs needed to cover the actual logistics.

    When asked about printing electronics, Herman said this is not yet available. No wonder, I thought, since mass chip producers like Samsung or Micron run billion dollar businesses to further shrink the transistors and optimize the complex production processes. That seems to be out of reach from the affordable printing business although Herman suspects it is not far away – perhaps a matter of a couple of years.

    Still, printing electronics actually got me thinking about the possibilities and development paths. I still remember the college classes where we’d be coding an 8051 circuit board with what could be designed in VHDL. I remember how painful it was. Trying to focus on the solution yet keep coming back to the strict rules of low level programming. I can imagine that the big players in electronic manufacturing can simply afford all the low-level programmers, for whom the development process simply became less painful over the years of practice. How would affordable 3D printing change this process? Would people like Herman’s R&D team be able to commoditize and popularize electronics manufacturing like it is possible today with mobile phone apps, where you can now create applications with limited programming knowledge? Imagine you’d be able to have a working board from just the logical design of your new solution.

    Today I am looking at the 3D printing business with a different perspective. I believe a bright future lies ahead.

    A glance at modern Information Security

    Who hasn’t wondered if they have chosen the right career path? During some self analysis recently, one of the areas I keep considering for myself is the Information Security path. Especially the position of CISO – after all I have a fair share of security engineering (not commercial) experience and some sense for organizing things. A recent article [1] describing the discovery of new feature of prime numbers got me thinking what sort of implications this could have and what I would do as a response. This is, after all, a rather forgotten plane, of all the planes and vectors the CISOs currently have to deal with.

    So, what are the IS challenges currently popular and how is this different?

    Let’s put the organizational part aside, including policies and procecures or dealing with social engineering. In technology, without any particular order, there’s the software security, with the most prominent type of vulnerability being related to memory management. I’m sure everyone has heard or read about exploiting buffer overflow scenarios. That is most of the time a result of lack of control of data sizes and the memory allocated for them. So, why not simply control it? That is because the circumstances required for a buffer overflow potential are not easy to notice when analyzing the source code. Something that is hard to spot and yet can lead to someone taking over control of a system puts this type of threat very high on the risk map. The good news? There are also many approaches to address vulnerabilities of this type, such as ASLR[2] or code audits.

    Another endangered area is again related to memory, but from the hardware perspective. Rowhammer is much less known, probably because a successful exploitation of this vulnerability “on a large scale” has never made it to the news – or even, above white paper [3] level. Nevertheless this threat is probably just as “red” on the risk map as the one above. Successful exploitation can lead to privilege escalation but the likelihood – well, that’s every PC and server memory currently used, including the new DDR4 [4]. There are hardware responses to this coming in new CPU instruction sets [5], however they are flawed in the typical way security solutions do – there’s a negative impact on speed/latency. The speed of computer memory has always been a bottleneck so in many implementations such solutions will not be acceptable.

    Speaking of hammers, there’s always the “hammer way” (as opposed to the clean-cut “scalpel way”) of exploiting a vulnerability. I am referring to the next security area commonly referred to as Denial of Service. The point of DoS is “if you cannot take over the control, at least stop the service” by flooding with spurious service requests. The financial implications can be quite severe. There are typical DoS flaws that can be addressed with smart programming, but also attacks based on distribution of source of requests (DDoS), against which there is no efficient solution, maybe aside unreasonably increasing the infrastructure footprint and some smart collaboration with ISPs.

    Last plane I wanted to cover here is cryptography. This probably is the easiest area – mathematicians define ciphers which can be used to encrypt communication. Those are coded in functions which then are packet into security libraries to be commonly used by programmers as required. Since proving a cipher to have weaknesses can take time after initial announcement, every now and then the IT world is rocked by a cryptography based vulnerability such as Heartbleed, weaknesses around MD5, or previously RC4 and so on. Nevertheless, the ciphers are in vast majority based all on the prime numbers theorem – or more importantly, that it is difficult to timely check if the exponent of a number is prime or not, using currently available efficiency of computation.

    In reference to [1], the typical weaknesses of ciphers, aside from quantum computing, have gained a new enemy. If there is a higher chance that for a prime number, the next digit in the next number is known by likelihood of 65%, then that should make “guessing” the number much easier. For example, one could assign priorities for the numbers when running a brute force check, thus speeding up the cracking process. This likelihood then is a clue, which can be used in a similar way you’d be cracking a password hash with tools like John The Ripper – by building a set of rules such as “passwords with numbers usually are constructed with the letters preceeding the numbers, i.e. password ‘Forthequeen96′”.

    I hope I am wrong about this, however from the theory it appears that with this discovery[1], all modern cryptography got another impulse to find a new, quantum-computing-proof cipher. If I am right, this discovery just makes all cryptography weaker, meaning – if you needed a super computer to crack encrypted communication, now you’d need less.

    1. https://www.quantamagazine.org/20160313-mathematicians-discover-prime-conspiracy/
    2. https://en.wikipedia.org/wiki/Address_space_layout_randomization
    3. http://users.ece.cmu.edu/~yoonguk/papers/kim-isca14.pdf
    4. http://arstechnica.com/security/2016/03/once-thought-safe-ddr4-memory-shown-to-be-vulnerable-to-rowhammer/
    5. http://blogs.cisco.com/security/mitigations-available-for-the-dram-row-hammer-vulnerability

    Simple SPAM solution with auto-learn and recovery options

    Perhaps you’ve come across those fancy, branded solutions that allow quarantining mail messages, delivery on demand (when you click a link of a suspected message), filtering incoming messages and so on. Upon some reseach it appears that these come with a price tag of over $60,000 in typical configurations. Expensive, huh? This got me thinking how hard would it be to deliver such functionality using open source solutions that come under the GPL license. Well, I’m a bit rusty now but every now and then try to do something like this, for mental hygiene. And it did not take me more than a couple of hours so it is simple and minimalistic.

    Side note: generally I think off the shelf solutions most of the time lack the flexibility while when it comes to their advantages, more often than not it is the packaging, branding and some poor fellow on the on-call support ready to answer basic questions. On the other hand, with the right team you can do wonders using GPL tools.

    For the purpose of presenting the system in this article, I will assume you were able to set up your server(s) with an MTA, clamav and spamassassin using whatever configuration you need to efficiently deliver email to your customers – internal or external. For me – I use exim, spamassassin and clamav, but you could do any other MTA as you prefer.

    Usually an MTA and spamassassin allow spam filtering by default using a collection of rules in Perl. On top of that, optionally you can enable an AI based on the Bayesian learning algorithm. Such setup all together offers the following functionality:

    • Unsolicited mail gets filtered out to some extent by the Perl rules which look for certain combinations of words and score penalty points.
    • Other messages can be learned by the algorithm to allow improving the efficiency of the filter

    Something is missing, right? For example:

    • You need manual intervention in case the filters missed a spam message
    • Nothing handles false-positive hits

    This guide will explain the most minimalistic method I could think of for delivering this missing functionality using open source solutions. Lets start with the easiest one – auto-learn without admin intervention.

    Enabling auto-learn per maildir folder

    This will allow me to further train the filter to reach decent values. When the filter is well trained, I was able to achieve efficiency of 100% for many months, while having processed an average count of up to 1000 messages per day.

    In this setup, I use maildir format for storage so that each IMAP folder is a filesystem folder and each message is a file. This allows me to announce and create for every user I support a special folder called SPAM directly inside the INBOX. As a part of the design, I can then ask users to move all files they consider spam to that folder and have a script that will periodically (use your favorite implementation of cron, I use cronie) scan these folders for new messages, learn them as spam and delete.

    Additional ideas – you can make this smarter by:
    1. Adding more controls to prevent mistaken drag-and-drop-and-forget.
    2. Move instead of delete, to some quarantine folder.

    A script performing such activity can look like this:

    #!/bin/zsh
    #
    # Learn messages thrown to the SPAM folder as SPAM and delete them
    # Script by Patryk Rzadzinski patryk@rzski.com
    #
    
    # Configurables
    maildir="put path to your maildir here";
    spamdir=".INBOX.SPAM";
    log="/usr/bin/logger -t spamd";
    
    # For whom should this run? Fill this the array with all your users
    mailers=(Alice Bob Charlie)
    
    for user in "${mailers[@]}"; do
    	if [ -d "$maildir/$user/$spamdir" ]; then
    		for spam in $(find "$maildir/$user/$spamdir" -type f ! -name "dovecot*" ! -name "maildirfolder"); do
    			$(which sa-learn) --spam "${spam}" 2>&1 >>/dev/null && eval "${log}" "Learned ${spam} message as spam.";
    			rm -f "${spam}" && eval "${log}" "Deleted ${spam}.";
    		done;
    	fi;
    done;

    Since this is a new thing, I have enabled simple logging using the logger tool available on any Linux system. The tag used here is called “spamd” which is arbitrary, however I have previously configured my syslog-ng to catch all system messages using this tag and keep 30 days of files in a specific folder. This might come in handy for debugging, but is not the subject of this article.

    Naturally, we need a crontab entry for this to work. How often should this script run? I think it depends on your load and amount of users. For home solutions you can have this run every minute, with the benefit of acting quickly and downside of someone moving a file by mistake, since there will be little time to react. I would recommend to start with hourly runs and then fine-tune as required.

    Allowing the user to deal with false positives

    One thing I really wanted at some point is to make sure an important email does not simply get filtered out. At the same time, I wanted vast majority of the spam that reaches my mailbox to simply disappear. I designed the following ompromise. A weekly digest of all the messages that were not delivered to me because the system considered them as spam (in other words, they scored a sufficient amount of penalty points) plus an option to recover each such message with 1-2 clicks. To achieve that, I wrote the following script that I have cron run for me once a week to generate a report about messages that were not delivered to me but thrown into the spam directory instead.

    #!/bin/zsh
    #
    rcpt="your@email.here";
    spamdir="path to your spam folder";
    
    cd $spamdir;
    spam_report="${spamdir}/workdir/temp;
    rm -f ${spam_report};
    n="<br>";
    recovery="<a href=\"mailto:RecoverSpam@yourdomain.com?subject=";
    
    # Removal of the base64 encoded text
    Mail_decode () {
            decoded="$(echo ${1} | cut -d' ' -f2)";
            if [[ "${decoded}" =~ "UTF-8" ]]; then
                    stripped="$(echo "${decoded}" | sed -e 's:=?UTF-8?[BQ,]?::g' -e 's:?=::')";
                    decoded="$(echo "${stripped}" | base64 -d)";
            fi;
            printf "${decoded}$n";
    }
    
    # Generate labels from spam message headers
    for spam in *; do
            if [[ ! -d ${spam} ]] && (((($(date +%s) - $(/usr/bin/stat -c %Y ${spam}))) < 604800)); then
                    (printf "Message ID: ${spam}$n";
                    printf "RECV: $(grep -i received: ${spam})$n";
                    printf "$(grep -i from: ${spam} | tr -d '<>')$n";
                    printf "$(grep -i to: ${spam} | tr -d '<>')$n";
                    printf "Subject: $(Mail_decode "$(grep -i subject: ${spam})")";
                    printf "${recovery}${spam}\">RECOVER$n";
                    printf "=========================================$n";) >> ${spam_report};
            fi;
    done;
    
    
    if [[ -z "${spam_report}" ]] ; then
            echo "no new messages" | /usr/bin/mailx -s "spam-digest: no new messages" ${rcpt};
    else
        	cat "${spam_report}" | /usr/bin/mailx -a 'Content-Type: text/html' -s "spam-digest" ${rcpt};
    fi;

    This script goes over the messages and in case they came over the last 7 days, it collects the information that would allow me to distinguish them from actual spam and send over a summary. It also adds a HTML link which allows me to recover such messages. Last but not least, I change the MIME type of the message in the report to HTML to allow processing of the a href part by an MUA – this allows the solution to work from roundcube and mail clients on my phone or laptop. I think it will also work with pine or mutt.

    In any case, the script results in the following report:

    ID: q1aYxhW-74796
    Envelope: (envelope-from foo@bar.com)
    From: Contact
    To: info infoaabbrzadzins.info
    Subject: Re: Information
    RECOVER
    =========================================
    ID: q1aYzMJ-74797
    Envelope: (envelope-from x@y.net)
    From: =?UTF-8?B?SmVycm9sZCBTb3Rv?=
    To: =?UTF-8?B?cGF0cnlr?= Reply-To: =?UTF-8?B?cGF0cnlr?=
    Subject: Free watches
    RECOVER
    =========================================
    ID: q1aYzTR-74849
    Envelope: (envelope-from z@asd.org)
    From: someone trying to send spam
    To: =?UTF-8?B?cGF0cnlr?= Reply-To: =?UTF-8?B?cGF0cnlr?=
    Subject: Inheritance
    RECOVER
    =========================================
    (...)

    This way at a glance I know what hit the spam bin. Each “RECOVER” text is actually a HTML link pointing at an email address using the mailto: directive. Such directive on most systems is configured to open a new message by the MUA the user (or the corporation) has chosen. It takes additional parameters. In my example, I set the Subject to the value equal the file name of the spam message I want to recover. When I click (or tap) on RECOVER under the message that I consider not to be spam, it will open a new “compose email” window with pre-defined subject and send it to a special mail address configured on my system, which will then trigger a script re-delivering the falsely-spambinned message to the original recipient.

    How to configure exim to understand and process such messages? This can be done on ACL level, using the “run” command, but I keep such functionality in the system filter. I use the following logic: when a message comes from an address that has authenticated (make sure you deny non-authenticated submissions), to the pre-defined RecoverSpam recipient, execute a shell script with the following 2 arguments: name_of_file (we pass that in the subject, right) and original sender (so if I want something recovered, the system sents back the message to me – this is in envelope-from header field).

    Here’s an example of exim system filter configuration:

    # Exim filter
    logfile /var/log/exim/filter.log
    
    if "$h_to:" contains RecoverSpam@rzski.com then
    	pipe "/path/to/spamrecovery.sh \"$h_subject:\" \"$sender_address\""
    	finish
    endif
    
    # regular spam action depends on the custom header line injected by the ACL
    if $message_headers: contains "X-ACL-Warn: message looks like spam" then
    	save	/path/to/spamdir 0640
    	finish
    endif

    Very simple and minimalistic, which was the choice set in the beginning of this article.

    If you have a CISO looking at all this, then you might want to secure this a bit more. The easiest would be adding more checks. One nice idea that comes to mind is adding soft tokens which could come from any system daemon based on, say, normalized 8-character strings from /dev/urandom - these can be injected in headers of the messages, you can define whatever you want and then have the script check their validity when processing the message for recovery. Be careful in larger installations, where you might run out of entropy. In such cases you could simply install a larger token issuing daemon.

    The last element of this system is the script that would process the recoveries. Again, this is the simpliest approach which can be nicely expanded with additional security and sanity checks as needed.

    #!/bin/zsh
    #
    # Recover files from spam, by Patryk Rzadzinski patryk@rzski.com
    
    # Configurables
    spamdir="/path/to/spam";
    mailer="/usr/sbin/exim";
    log="/usr/bin/logger -t spamd";
    
    # Change this to match your spam tag or token
    clean="/bin/sed -i '/X-ACL-Warn: message looks like spam/d'";
    
    userid="${2%\@*}";
    
    # Conditioning - tighten as required.
    # Replace the last check with whatever authentication method you use - this one is for passwd
    if [ ! -z "$1" ] && [ "$#" -eq 2 ] && [[ "${userid}" =~ ^[a-zA-Z]{4,}$ ]] && id -u "${userid}" >/dev/null 2>&1; then
    	# remove the spam identificator from the message
    	eval "${clean}" "${spamdir}/${1}" && eval "${log}" "Removed spam tag from message $1.";
    
    	# teach spamd that this message is not spam
    	$(which sa-learn) --ham "${spamdir}/$1" 2>&1 >> /dev/null && eval "${log}" "Spamd learned message $1 as ham.";
    
    	# re-deliver
    	cat "${spamdir}/${1}" | eval "${mailer}" "${2}" && eval "${log}" "Re-delivered message $1 to recipient";
    
    	# delete message from spamdir
    	rm -f "/var/spool/spam/$1" && eval "${log}" "Removed message file $1 from ${spamdir}.";
    
    	# return 0 to confirm script completed successfully
    	exit 0;
    else
    	printf "Arguments passed to spam recovery are not OK - 1: $1 2: $2 3: userid: ${userid}.\n";
    	exit 1;
    fi;

    As before, there is again space for improvement. For example, instead of a single RECOVER “button”, you could actually make it optional depending on arguments, and offer the user multiple choice of actions: delete, deliver, learn as HAM and deliver, etc. In my case, RECOVER means (in this order): remove the spam tag AND learn as HAM AND re-deliver to original recipient AND remove from the spamdir. This is simple but can be improved for flexibility. Even better if you expand the mail-digest script with some branding to make your users happy and give them the “expensive solution” feel, add some graphics.

    And that’s all! Note that you could move this whole logic to PHP and process it over the web, but it does then make it much less minimalistic and you end up securing PHP, which is always fun and full of pitfalls.

    Another thing worth doing is moving the recovery process to a Docker container running only exim and sharing the spamdir folder, which would be a nice piece of sandboxing. I did not do this only because my VPS is low on disk space, but this solution should make CISOs happy and also offer some peace of mind.

    Comment on vision & strategy in light of the Apple vs FBI case

    Originally I published this on LinkedIn, here’s the comment.

    The [lost] ability to define long term [IT] strategy

    One thing I absolutely love about the US presidential elections is that the candidates are actually challenged to provide opinions on theoretic subjects that actually matter. This allows getting a decent insight into their ability to define long term strategies often on subjects which are very abstract and require a certain type of mental discipline and the ability to imagine multiple levels of implications. The most recent example is the case of the demand against Apple to introduce backdoors to their software.

    Long story short, those candidates remaining in the race, with an exception from the Libertarian Party, agree that the government needs to have some sort of a “master key” in order to tap into communications where required [1]. As commenters [2] already noticed – such requirement leads to a number of issues. First of all, the next vendor might not be US-based and thus the whole efort will be futile. Second, currently the suspects have used software as provided by the vendor, but what stops them from creating a cryptographic application that would encrypt the communication for them? Nevertheless, the presidential candidates make it quite clear in their speaches: security above liberty, because “something must be done”.

    So that leads me to the long term part. By the rule of induction – should cryptography be banned alltogether? No ciphers, all communication in the open? I think it is safe to say it is clear such approach will just not work. In vast majority of cases cryptography is there to secure our information, payment card information inclusive.

    So the residual facts that remain seem to be:

    • Backdoors or Master Keys are not the answer – they do not solve the problem, but are probably likely to win some votes. And they end up opening a whole new bag of problems
    • Cryptography is here to stay.

    But the long-term question stands: what should we be doing to avoid these situations? And how does all this tie to IT? In my line of work, I often lead technical incident response teams challenged to find a solution to an actual problem. One thing I have learned over the time is that sometimes having the best minds in the team is simply not enough to solve a case. Sometimes you need to take a few steps back and realize the root cause is outside of the picture everyone is focusing on. Sometimes, a long term strategy of doing (or not) certain things in a certain (standardization!) way offers the solution – the catch is, it might seem completely unrelated!

    Does that mean the FBI should just allow San Bernardino to happen? Of course not, simply the root cause is completely outside of the scope of the discussion and cryptography has nothing to do with it. The problem will not be solved in this area, but who will be the leader that can still notice that and can democratic elections still give us such leaders?

    1. http://windowsitpro.com/security/where-do-presidential-candidates-stand-encryption
    2. http://politics.slashdot.org/story/16/02/19/0019218/where-do-the-presidential-candidates-stand-on-encryption

    Jack and ALSA: sound through multiple devices

    I have designed a simple audio-video solution for my entertainment room at home, where I have a:

    • High quality headphones connected to a decent pre/post amplifier which connects to the computer through USB (and registers as USB audio device with my Windows10 and Gentoo linux).
    • Not so high (but not bad at all!) on-board sound chip provided with my motherboard – this one I’d like to connect to my TV.

    The goal is to have sound via the on-board card to the TV for less demanding tasks, such as watching movies and have sound go via USB to the amp while when I listen to music.

    After some reseach it appears ALSA on its own cannot do this job, but Jack can be used. At first I did not like the overhead, but quickly realised it is minimal. The below howto is based on JACK. On the 4.3 kernel (no -rt stuff since most of the good work for real time processing is long in the mainline kernel), it works well.

    First of all it is good to ensure all packages are build with jack support (and USE flags in Gentoo make this a trivial task) and install the Jack daemon (jackd). Now, for some reason it is a daemon that does not have a startup script, so you’ll need to figure out how you want to start it – for me it ended up in the xfce session controller (so it starts with my user ID when I log in) – but you can write a script too, just ensure it depends on alsasound.

    There is a gotcha here – for running the jackd without root privileges, you need to edit limtis.conf to give the audio group rt permissions:

    Code:
    root@ryba ~ # grep audio /etc/security/limits.conf
    @audio      –  rtprio          99
    @audio      –  memlock      unlimited
    @audio      –  nice      -10


    And of course, have your user in the audio group. As always with groups – for them to take effect, all sessions must be finished and re-started.

    Now, trick is you still need alsasound daemon running and a proper /etc/asound.conf which mixes multiple devices. Basically the following config works for the first two sound cards, because it uses the indexes instead of the device names (you get those from aplay -l. An exception is with the last config section, because it is apparently a dummy control, alsa needs it):

    Code:
    root@ryba ~ # cat /etc/asound.conf
    pcm.both {
    type route;
    slave.pcm {
    type multi;
    slaves.a.pcm “plughw:0,0”
    slaves.b.pcm “plughw:1,0”
    slaves.a.channels 2;
    slaves.b.channels 2;
    bindings.0.slave a;
    bindings.0.channel 0;
    bindings.1.slave a;
    bindings.1.channel 1;
    bindings.2.slave b;
    bindings.2.channel 0;
    bindings.3.slave b;
    bindings.3.channel 1;
    }
    ttable.0.0 1;
    ttable.1.1 1;
    ttable.0.2 1;
    ttable.1.3 1;
    }pcm.jack {
    type jack
    playback_ports {
    0 system:playback_1
    1 system:playback_2
    }
    capture_ports {
    0 system:capture_1
    1 system:capture_2
    }
    }

    pcm.!default {
    type plug;
    slave.pcm “jack”;
    }

    ctl.!default {
    type hw;
    card 0;
    }

    Next, I need to tell jack to start WITH this config too, so my line that starts it under xfce session starter is as follows:

    Code:
    /usr/bin/jackd -d alsa -P both

    Explanation: run jackd using -Device alsa and -Playback both – because that’s how my device is named in /etc/asound.conf: pcm.both

    Final step is to tell your software, such as mplayer, moc music on console, or whatever else you use, to switch from alsa to jack – that’s fairly easy, just rtfm, however you can also leave it as alsa, since this one is routed into the jack as well, making jackd the catch-all solution. It is quite important to do so because if you use Firefox and would like sound from flash and HTML5, FF will first try pulseaudio and if it is not present (like in my case, because I don’t want that overhead), then it will try alsa – but not jack and there is no config handle for that. So the above config basically uses jackd every time something tries to use alsa.

    Needed: alsa-plugins, jack-audio-connection-kit
    Not needed: alsa_out, qjackctl (the gui thing), gst-plugins-alsa, pulseaudio