Hopefully, this is old news for most of you reading this but I just ran into it and thought I'd share. Apparently IOS 12.2(58) and higher have major memory leaks on 2960's and 3750's. After rebooting to the new IOS, the switch has so little free memory that you can't even gain access to it via SSH, telnet or even console. The only option is the break the boot up sequence and boot to an older image (assuming you didn't delete it already). Check details: here
So, I figured I'd go with 12.2(55) but alas that has a bug with dot1x multi-auth (a feature I very much rely on)
So, be careful out there - apparently all the really good QA guys over at Cisco have retired....
Monday, December 5, 2011
Wednesday, July 20, 2011
Cisco ASA: ERROR: Authentication Rejected: Memory error
I was setting up a new ASA this morning and ran into a strange error when attempting to test my LDAP servers. After many google searches, I discovered there isn't alot of information about this error out there. I found one post that mentioned an incorrect base DN for searches caused the error for him but this wasn't the case for me. I have several ASA's already in service and had copied and pasted the config for the LDAP servers from an existing firewall. I finally figured out that I needed to re-enter the password either in ASDM or CLI because the encrypted value that I pasted in was not being interpreted correctly in the new firewall. I'm guessing because the general purpose RSA keys are different between the two firewalls. In any case, I figured this out the hard way so I thought it might save you guys some time!
Paul
Paul
Thursday, July 7, 2011
The life of an email.
One of the best tools I've used over the years to learn something new is to go through the entire process of how it happens in EXTREME detail from start to finish. In today's post, I'll walk you through how a typical corporate email is sent and ultimately reaches its destination in as much (relevant) detail as I can. For example, I will explain where a DNS query is used but will not discuss the intricacies of how DNS works and what would happen if the primary DNS server didn't respond, etc. in order to stay on topic.
In order to create the most relevant information I can, I'll try to keep things general by avoiding any specific applications.
1) A user sitting a PC in a corporate network types an email from wiley@acme.com into their mail client to bob@contoso.com and clicks 'send'.
2) The mail client will attempt to send the mail to the preconfigured mail server. Most often, the mail server is configured with a DNS name.
3) The PC looks in its DNS cache for the mail server. If the IP is not in cache, the PC send a DNS query to its primary DNS server to obtain the IP address of the mail server.
4) Now that the PC has the IP of the mail server, it will start to think about making a connection to it. Therefore, it looks at its IP address and subnet mask to see if it is in the same subnet as the mail server. If it isn't, it checks its local routing table ('route print' on windows) to see if there is a specific route to the mail server. If not, it will send the packet to its default gateway. (It followed this same process in order to contact the DNS server in step 3 too by the way)
5) The PC now knows where to send the packet so it creats a new connection to the mail server and sends the email to the default gateway who routes it to the mail server.
6) The mail server now has the task of delivering the message. The server might deliver mail itself but more likely, it will forward Internet-bound email to another device that will check the message against security policies for things like viruses, spam, data loss prevention, size, content, it may add a disclaimer to the message, etc.
7) After checking out ok against policies, the message needs to be delivered. The server sending the message knows that the destination is bob@contoso.com but it needs to know the IP address of the server where it needs to send the message. It performs a DNS query using root hints to get a list of .com DNS servers. Root hints servers are at the highest level of the DNS heirarchy on the Internet. Root Hints servers know which servers handle .com, .net, .org, .etc. Those servers then know who handles hotmail.com, google.com, and so on and so forth. Getting back to the task, the corporate DNS server now has a list of .com DNS servers. It then sends another query to a .com DNS server looking for contoso.com and gets a response of Contoso's public DNS server. A third DNS query is now sent to Contoso's public DNS server asking for MX (Mail Exchanger) records for the contoso.com domain. The contoso.com DNS server responds with those records and any preferences assigned. Finally, the corporate DNS server replies to the mail server with the IP addresses of contoso.com's mail servers. Phew!
8) The mail server at acme.com chooses the MX record with lowest preference and opens a connection to TCP port 25 on that server. This connection must be allowed to traverse firewalls at both acme and contoso.
9) Depending on the type of anti-spam device that is in use at contoso, several checks can now take place before the message is even sent. Contoso's Antispam device can check the reputation of the IP address that is sending the message (Acme's public IP) against databases such as senderbase, spamhaus to see if it is a known spammer. If there is a match, the antispam device will most likely end the connection. The idea behind all these checks is to verify that the sender is not a spammer and that the message did in fact come from the actual sender rather than being spoofed (forged). So, if the connection is ok so far, acme will send contoso a helo with a greeting that identifies itself as mail1.acme.com. Contoso might then perform reverse greeting and reverse DNS checks. In a reverse greeting check (hotmail does this), contoso will perform its own DNS queries to look up mail1.acme.com and will open a connection to that server. It will then compare the greeting it receives from this known correct server to the greeting it was given eariler since the sending box is claiming to be mail1.acme.com. If the greetings don't match, mail can be rejected, throttled and your senderbase score will drop. Next, a reverse DNS check is performed. The mail server at acme.com performs a DNS lookup on the IP address with which is has just formed a connection. Reverse DNS is maintained by the Internet Service Provider for a given block of IP Addresses. The reverse IP should resolve to the name of the mail server namely, mail1.acme.com. If the reverse dns does not match, mail can be rejected, throttled and your senderbase score will drop.
10) Assuming acme.com has everything setup correctly and everything checks out, the server at contoso will respond with a 220 or "go ahead". The mail server at acme now sends the recipient information. Before proceeding any further, the receiving mailserver at contoso will then lookup the recipient to verify that he/she exists. If not, the connection ends.
11) Sender information is sent which again is compared with the sending domain to make sure there's no switching going on.
12) Next, size and content is evaluated by contoso's anti-spam products, the message is scanned for viruses, etc and is finally delivered to contoso's mail server and the recipient's inbox.
And all you had to do was click "Send"! :-)
I hope this has been informative and look for a post on troubleshooting this process in the future. Thank you for viewing.
Paul Ackerman
In order to create the most relevant information I can, I'll try to keep things general by avoiding any specific applications.
1) A user sitting a PC in a corporate network types an email from wiley@acme.com into their mail client to bob@contoso.com and clicks 'send'.
2) The mail client will attempt to send the mail to the preconfigured mail server. Most often, the mail server is configured with a DNS name.
3) The PC looks in its DNS cache for the mail server. If the IP is not in cache, the PC send a DNS query to its primary DNS server to obtain the IP address of the mail server.
4) Now that the PC has the IP of the mail server, it will start to think about making a connection to it. Therefore, it looks at its IP address and subnet mask to see if it is in the same subnet as the mail server. If it isn't, it checks its local routing table ('route print' on windows) to see if there is a specific route to the mail server. If not, it will send the packet to its default gateway. (It followed this same process in order to contact the DNS server in step 3 too by the way)
5) The PC now knows where to send the packet so it creats a new connection to the mail server and sends the email to the default gateway who routes it to the mail server.
6) The mail server now has the task of delivering the message. The server might deliver mail itself but more likely, it will forward Internet-bound email to another device that will check the message against security policies for things like viruses, spam, data loss prevention, size, content, it may add a disclaimer to the message, etc.
7) After checking out ok against policies, the message needs to be delivered. The server sending the message knows that the destination is bob@contoso.com but it needs to know the IP address of the server where it needs to send the message. It performs a DNS query using root hints to get a list of .com DNS servers. Root hints servers are at the highest level of the DNS heirarchy on the Internet. Root Hints servers know which servers handle .com, .net, .org, .etc. Those servers then know who handles hotmail.com, google.com, and so on and so forth. Getting back to the task, the corporate DNS server now has a list of .com DNS servers. It then sends another query to a .com DNS server looking for contoso.com and gets a response of Contoso's public DNS server. A third DNS query is now sent to Contoso's public DNS server asking for MX (Mail Exchanger) records for the contoso.com domain. The contoso.com DNS server responds with those records and any preferences assigned. Finally, the corporate DNS server replies to the mail server with the IP addresses of contoso.com's mail servers. Phew!
8) The mail server at acme.com chooses the MX record with lowest preference and opens a connection to TCP port 25 on that server. This connection must be allowed to traverse firewalls at both acme and contoso.
9) Depending on the type of anti-spam device that is in use at contoso, several checks can now take place before the message is even sent. Contoso's Antispam device can check the reputation of the IP address that is sending the message (Acme's public IP) against databases such as senderbase, spamhaus to see if it is a known spammer. If there is a match, the antispam device will most likely end the connection. The idea behind all these checks is to verify that the sender is not a spammer and that the message did in fact come from the actual sender rather than being spoofed (forged). So, if the connection is ok so far, acme will send contoso a helo with a greeting that identifies itself as mail1.acme.com. Contoso might then perform reverse greeting and reverse DNS checks. In a reverse greeting check (hotmail does this), contoso will perform its own DNS queries to look up mail1.acme.com and will open a connection to that server. It will then compare the greeting it receives from this known correct server to the greeting it was given eariler since the sending box is claiming to be mail1.acme.com. If the greetings don't match, mail can be rejected, throttled and your senderbase score will drop. Next, a reverse DNS check is performed. The mail server at acme.com performs a DNS lookup on the IP address with which is has just formed a connection. Reverse DNS is maintained by the Internet Service Provider for a given block of IP Addresses. The reverse IP should resolve to the name of the mail server namely, mail1.acme.com. If the reverse dns does not match, mail can be rejected, throttled and your senderbase score will drop.
10) Assuming acme.com has everything setup correctly and everything checks out, the server at contoso will respond with a 220 or "go ahead". The mail server at acme now sends the recipient information. Before proceeding any further, the receiving mailserver at contoso will then lookup the recipient to verify that he/she exists. If not, the connection ends.
11) Sender information is sent which again is compared with the sending domain to make sure there's no switching going on.
12) Next, size and content is evaluated by contoso's anti-spam products, the message is scanned for viruses, etc and is finally delivered to contoso's mail server and the recipient's inbox.
And all you had to do was click "Send"! :-)
I hope this has been informative and look for a post on troubleshooting this process in the future. Thank you for viewing.
Paul Ackerman
Monday, July 4, 2011
The WinKit revived?
About 6 years ago, I spent several months developing a suite of network administration tools called the WinKit. Here's a brief description of each tool.
AdminPW:
CheckUser:
FileCheck:
FileCopy:
NTFS Auditor:
The original code for the WinKit was written in VB6.0 and I haven't had the time to learn and convert it to .NET. The app does run on Windows 7 although there are a few bugs and it has been tested with a 2008 R2 domain at R2 functional level. If anyone has an interest in updating the toolkit for market, please get in touch with me. PaulAckerman1000 'at' hotmail
AdminPW:
This is a very powerful program that will allow a single technician to change the local administrator password on hundreds of computers at once with the click of the mouse. Complete with an input file to read computer names and an export file for any unreachable computers as well as Active Directory integration, AdminPW won’t miss a beat. The export file can even be used again as input to make sure all the computers are completed in the list.
It has always been a time-consuming process to go around to each computer on the network and change passwords when a technician or other member of IT leaves the organization. This often results in passwords not being changed at all creating a large security risk in your network. It is common knowledge that most security breaches occur from within by current or former employees. With AdminPW, it will only take you a minute to secure your network by changing the admin passwords on ALL your workstations! AdminPW alone makes the WinKit a “must-have” for any network.
CheckUser:
Wouldn’t it be nice for helpdesk to be able to pop a username into a program and have it tell you all the computers on the network where that user is logged on? CheckUser can do just that. After you enter the username, CheckUser will tell you the computer names where that user is currently logged in or was the last person to logon to that computer. It can also work backwards. You can enter a computer name and in less than a second find out who is currently logged onto that computer. If no one is logged on, you can find out who was the last person to log on! This is great for security when you need to quickly find out who is logged on a specific computer or where a specific username is active. As with the other applications in the WinKit, they can be as useful as you make them for many different situations.
FileCheck:
FileCheck is a useful tool that can check a group of computers on a network for a specific file on their local harddrives. The usefulness of this application is limited only to your imagination. For example, to build a list of all the computers on your network where a certain application is installed such as Microsoft Access, simply enter a filename that application uses (“msaccess.exe”) and click search. Another use is to quickly scan for viruses on the network that create specific files during outbreaks. FileCheck is by no means similar to searching for a file with Windows search which can take minutes to find a file. FileCheck runs very quickly. Since you specify the exact path of the file you are searching for, it can connect and check several computers per second.
FileCopy:
Another tool, like FileCheck, that can actually copy files from a source on the network to destination computers. You can copy a single file or multiple files and folders / subfolders.
For example, you need to update an Oracle tnsnames.ora file on computers so you plan to overwrite it on all computers on the network immediately. Simply enter the source path for tnsnames.ora and then click Start! Depending on the size of the source files, FileCopy can update hundreds of computers with the new files within minutes. No need to wait for logon scripts and you don't have to worry about those people that never reboot. FileCopy's changes happen in real time.NTFS Auditor:
Here’s a hypothetical situation. The CEO just told you that he is concerned that an individual has been accessing the shared group folders on the file server. He wants to know exactly what folders and files that person has access to right now. How can you audit a specific user without manually checking permissions on each folder on the server? I’m sure there have been several times when you’ve needed to know exactly which folders and files on your file server a certain user has access to. With NTFS Auditor, simply map a drive to the file server you want to audit and enter a domain/username or domain/group that you are checking permissions for and click “Execute”. The program will build a text file for you containing each folder and file that user/group has access to on the server. You can also simply browse a computer or mapped drive to instantly see the NTFS permissions on that folder. Simple results are easy to understand. **Microsoft has caught up and this functionality now exists in Server 2008!!
The original code for the WinKit was written in VB6.0 and I haven't had the time to learn and convert it to .NET. The app does run on Windows 7 although there are a few bugs and it has been tested with a 2008 R2 domain at R2 functional level. If anyone has an interest in updating the toolkit for market, please get in touch with me. PaulAckerman1000 'at' hotmail
Top 5 things wrong with network security
Happy 4th of July!!!
To celebrate this Independence Day, I've decided to write an article to help you gain independence managing your network security. So, in an effort to help tighten up commonly seen errors and blunders in networks today, I've devised this list of the top five things wrong with network security today.This type of a list can be built using a multitude of criteria and considerations. In this discussion, we look at the items that aren't necessarily the worst kind of mistake but those that are likely to occur and have the largest negative impact on a company.
5. Domain Admin accounts for $1 / Pass-the-hash / not changing admin passwords. It seems like organizations are granting domain admin rights as part of the new user package these days. Ok, maybe it's not that bad but how many people are truly needed to manage the domain? Are you still granting helpdesk staff domain admin rights just so they can join PC's to the domain? When software vendors come in and says my product needs to be installed with a domain admin account with no reasoning and sometimes without an understanding of why they need those rights, ask them for the specific reason or try it with a standard user account and see what happens. Certainly never give domain admin accounts out to vendors or consultants. If they are working remotely, you can type in the password when necessary and avoid being part of a "Con-Net" (network similar to a bot-net consisting of machines/domains that are owned by a consultant). Administrative and service accounts need to be dished out cautiously and maintained. Changing service account passwords can be a nuisance but it is especially necessary when IT staff leave the company and should be done regularly for safe measure. Also, don't use accounts on your webservers that have rights to other servers in the forest. Webserver services and connections should have unique, dedicated, audited accounts with strong passwords (that should also be periodically changed). And for goodness sakes, don't use the same password for your local administrator that you use for the domain administrator account!
4. Applications. Both in-house and vendor-provided applications have the same problem - they usually aren't secure. Many application developers don't have a solid understanding of servers, networking and especially security. Especially for any applications that will be available over the Internet, work with your developers to incorporate security from the beginning stages of development. Security as an afterthought doesn't work well. Hire a team to perform an assessment of your application BEFORE it goes online. If there is a database behind the app, ensure it does live on the same server that your HR database does and that the user account the web server uses to communicate with the database has proper permissions, use read-only views, etc.
3. Rogue access points. Many organizations still have not deployed a wireless infrastructure for staff use. As such, when Bob wants to play on his ITouch/IPad/IPhone (yeah IPhone since AT&T's service stinks!), it suddenly seems like a great idea to go to BestBuy and pick up a $49 netgear wireless router and slap it on the network in between their PC and the wall jack. The problem is that this happens a lot more that you might think and can be difficult to prevent without spending a good amount of time and money on network access control at the switch. The problem is that now you have no control over the devices on the network. They are un-patched, have old Anti-virus definitions if they have AV at all and are going all over the internet because they think they aren't being tracked. Many OS worms infiltrate and propagate through networks this way. A really cheap method of dealing with this is to give your helpdesk staff WiFi detecting keychains. You know, the kind that beep and display the SID of a wireless network when one is detected. This is sort of an on-going detective control without costing anyone additional time or major expense. If you're serious about protecting your network from unauthorized access, consider standing up a PKI and deploying an access control server like a Cisco ACS 1120 and require certificate-based authentication on the switches with a guest vlan for internet access. This solution keeps the users happy while protecting the network.
2. Laptops at home. It's almost impossible for companies to do business anymore without having laptops at home. Unfortunately, while many of our telecommuters are somewhat tech savvy, they aren't security pro's which means they sometimes fall prey to simplifying the VPN process by leaving all usernames, passwords, instructions, etc in the bag with the laptop. The risk of laptops being stolen is not only real but it is significant. According to ADT, laptops and home electronics are among the most stolen items from homes second only to tools and cash. Two factor authentication is helpful. You could use tokens but that would only get left in the bag as well. My recommendation is to purchase laptops with fingerprint readers. These have become very accurate and are difficult for the average Joe to circumvent. The best part is that they don't really add any inconvenience to our already overwhelmed users. Encrypting laptop hard drives with BitLocker or something similar also goes a long way to protecting the data they contain.
1. Backups make the top of the list. There is nothing more important to your success than having a solid backup strategy. Not to be pessimistic, but we need to operate under the mindset that eventually, we will all fail in protecting our networks from disaster of one kind or another. Large companies are often infiltrated by attackers while smaller organizations are victims of fires, floods and miffed employees that have too many privileges on the network. Many companies simply don't have a backup strategy. Those that do often don't test the backups or worse - aren't really sure how to go about restoring the data they contain. Do know how to restore your Active Directory if needed? Backups are a necessity but they also introduce a security risk themselves. Are your backup tapes encrypted? If not, why not? Backups are one-stop shopping for someone trying to access your company data and can be highly sought after. In short, make sure you make backups that actually contain the data you will need when it comes time to restore. Make sure you know how to restore the data and test that process. Finally, ensure the tapes are encrypted and protected.
To celebrate this Independence Day, I've decided to write an article to help you gain independence managing your network security. So, in an effort to help tighten up commonly seen errors and blunders in networks today, I've devised this list of the top five things wrong with network security today.This type of a list can be built using a multitude of criteria and considerations. In this discussion, we look at the items that aren't necessarily the worst kind of mistake but those that are likely to occur and have the largest negative impact on a company.
5. Domain Admin accounts for $1 / Pass-the-hash / not changing admin passwords. It seems like organizations are granting domain admin rights as part of the new user package these days. Ok, maybe it's not that bad but how many people are truly needed to manage the domain? Are you still granting helpdesk staff domain admin rights just so they can join PC's to the domain? When software vendors come in and says my product needs to be installed with a domain admin account with no reasoning and sometimes without an understanding of why they need those rights, ask them for the specific reason or try it with a standard user account and see what happens. Certainly never give domain admin accounts out to vendors or consultants. If they are working remotely, you can type in the password when necessary and avoid being part of a "Con-Net" (network similar to a bot-net consisting of machines/domains that are owned by a consultant). Administrative and service accounts need to be dished out cautiously and maintained. Changing service account passwords can be a nuisance but it is especially necessary when IT staff leave the company and should be done regularly for safe measure. Also, don't use accounts on your webservers that have rights to other servers in the forest. Webserver services and connections should have unique, dedicated, audited accounts with strong passwords (that should also be periodically changed). And for goodness sakes, don't use the same password for your local administrator that you use for the domain administrator account!
4. Applications. Both in-house and vendor-provided applications have the same problem - they usually aren't secure. Many application developers don't have a solid understanding of servers, networking and especially security. Especially for any applications that will be available over the Internet, work with your developers to incorporate security from the beginning stages of development. Security as an afterthought doesn't work well. Hire a team to perform an assessment of your application BEFORE it goes online. If there is a database behind the app, ensure it does live on the same server that your HR database does and that the user account the web server uses to communicate with the database has proper permissions, use read-only views, etc.
3. Rogue access points. Many organizations still have not deployed a wireless infrastructure for staff use. As such, when Bob wants to play on his ITouch/IPad/IPhone (yeah IPhone since AT&T's service stinks!), it suddenly seems like a great idea to go to BestBuy and pick up a $49 netgear wireless router and slap it on the network in between their PC and the wall jack. The problem is that this happens a lot more that you might think and can be difficult to prevent without spending a good amount of time and money on network access control at the switch. The problem is that now you have no control over the devices on the network. They are un-patched, have old Anti-virus definitions if they have AV at all and are going all over the internet because they think they aren't being tracked. Many OS worms infiltrate and propagate through networks this way. A really cheap method of dealing with this is to give your helpdesk staff WiFi detecting keychains. You know, the kind that beep and display the SID of a wireless network when one is detected. This is sort of an on-going detective control without costing anyone additional time or major expense. If you're serious about protecting your network from unauthorized access, consider standing up a PKI and deploying an access control server like a Cisco ACS 1120 and require certificate-based authentication on the switches with a guest vlan for internet access. This solution keeps the users happy while protecting the network.
2. Laptops at home. It's almost impossible for companies to do business anymore without having laptops at home. Unfortunately, while many of our telecommuters are somewhat tech savvy, they aren't security pro's which means they sometimes fall prey to simplifying the VPN process by leaving all usernames, passwords, instructions, etc in the bag with the laptop. The risk of laptops being stolen is not only real but it is significant. According to ADT, laptops and home electronics are among the most stolen items from homes second only to tools and cash. Two factor authentication is helpful. You could use tokens but that would only get left in the bag as well. My recommendation is to purchase laptops with fingerprint readers. These have become very accurate and are difficult for the average Joe to circumvent. The best part is that they don't really add any inconvenience to our already overwhelmed users. Encrypting laptop hard drives with BitLocker or something similar also goes a long way to protecting the data they contain.
1. Backups make the top of the list. There is nothing more important to your success than having a solid backup strategy. Not to be pessimistic, but we need to operate under the mindset that eventually, we will all fail in protecting our networks from disaster of one kind or another. Large companies are often infiltrated by attackers while smaller organizations are victims of fires, floods and miffed employees that have too many privileges on the network. Many companies simply don't have a backup strategy. Those that do often don't test the backups or worse - aren't really sure how to go about restoring the data they contain. Do know how to restore your Active Directory if needed? Backups are a necessity but they also introduce a security risk themselves. Are your backup tapes encrypted? If not, why not? Backups are one-stop shopping for someone trying to access your company data and can be highly sought after. In short, make sure you make backups that actually contain the data you will need when it comes time to restore. Make sure you know how to restore the data and test that process. Finally, ensure the tapes are encrypted and protected.
Subscribe to:
Posts (Atom)