RSS and atom feeds are XML files. It would be possible to parse these files and structure the contents from scratch – but fortunately we don’t need to do that. The excellent module feedparser can do it for us. Here’s what’s done in our little reader :
Parse it: fp = feedparser.parse(feed.url)
fp is now a dict containing the feed data. The most important fields are fp.title and fp.entries
Woodscreaming.com is a traditional Django app with minimal frontend processing. Creating the list of feeds is simply done by passing parsed feeds to a Django template.
A term that is often used in the cybersecurity community is threat hunting. This is the activity of hunting for intruders in your computer systems, and then locking them out. In the more extreme cases it can also involve attacking them back – but this is illegal in most countries. Threat hunting involves several activities that you can do to find hackers on your network. The reason we need this is that the threats are to some extent intelligent operators who adapt to the defenses you set up in your network – they find workarounds for each new hurdle you throw at them. Therefore, the defense needs to get smart and use a wide arsenal of analysis techniques to find the threats; meaning analysis of data that can indicate that an intrusion has occurred. Data on user behavior, logins, changes to files, errors, and so on can be found in the systems logs. In addition to things that can be automated (looking for peaks in network traffic, etc.), threat hunting will always include some manual inquisitive labor by the analyst – both for understanding the context more deeply, and perhaps utilizing statistical and data science tools for special cases. Based on successful hunts, automated signals can be added to improve future resilience. The interplay between automated red flags, context intelligence and data science is shown below.
The conglomerate joint venture deal: a potential source of an advanced persistent threat?
Johnny the Hunter was going to work as usual in the morning. He got a cup of coffee at sat down at his computer to start his day. As most office workers, Johnny first skimmed his e-mails, and checked his Twitter feed for any interesting news. He noticed one e-mail that stood out, from one of the sys.admins, who told him that one of the application servers had rebooted without any good reason last night. No functionality had been lost, and no significant downtime was recorded – it was just a simple reboot. The logs on the server did not show any suspicious activity.
This triggered Johnny’s curiousity – what had casued the reboot? Was it some random hardware issue? Was it a software bug causing a kernel crash? Probably not, that would have been recorded in the server logs.
Johnny decided to make this the starting point for a hunt. First, he checked all automated surveillance systems; there were a few orange flags (detected abnormal activity but not something considered critical). He decided he needed to review the newest intelligence data they had on the threat landscape. There was nothing from the typical providers that caught his attention, so he turned to the intranet to check if something was going on internally in the company. He noticed the CEO had posted a video explaining that they were negotiating with an Asian conglomerate about buying up one of the conglomerate’s competitors as a joint venture. They had not yet agreed on who would be the controlling company in the joint venture. He didn’t notice any other big news.
He then called HR to ask if there were any new hires onboarding that would have anything to do with the Asian deal. The HR director told him that they had several applicants, all coming from the Asian conglomerate, and they were all highly qualified. It seemed a waste of talent not to hire at least one of them but the CSO had told HR to hold it off.
Johnny decided to start looking at network logs from the last 2 years, to have a baseline, and then to look for anomaly’s after negotiations about the buy-up started. For this he collected logs not only from the application servers, e-mail servers, web servers and network security devices, but also news items and social media posts. He deciced he would use supervised learning to correlate news events with network anomalies and called up Sin Jing, the head of their internal big data and machine learning R&D unit to discuss how best to do this.
Using a range of techniques Johnny investigated behaviors and could find a correlation between news and strange network activities from the last 4 months. Prior to that there was no such correlation. He also tracked down the activity to two user accounts in the accounting department, and the activity was always managed over VPN outside of normal office hours. He had a lead on the threat actors – and decided to discuss it with the HR department to assess the possibility of this being an insider threat, or if the compromised accounts were simply compromised accounts not detected by their endpoint security solutions.
This is threat hunting – and for the most advanced threats it is the only way to decrease detection time, and to effectively reduce the attack surface.