Tag Archives: ISP

Lessons learned from being a Systems Administrator for an ISP

Lessons learned from being a Systems Administrator for an ISP

As one of my first forays into entrepreneurship, I co-founded an Internet Service Provider with a friend of mine. I still own the ISP and somewhat run it (it’s no longer my “day” job but I still get to tinker with the infrastructure…I have a couple of really good employees that run the daily operations). Looking back, there are a lot of things that, if done differently, would have saved me many hours of headaches and lost sleep. I’ve attempted to compile a list of some of those items, both for my own future reference and hopefully to help anyone else that may be setting out to build or maintain a consumer oriented publicly-facing service provider network.

  1. Lock down outbound port 25 (SMTP) connections to only allow access to your own servers. If any of your access customers are using outside e-mail services, have them contact the admin for that mail provider or service; they likely offer an alternative port (such as 587 or 2525, or just enabling SSL for the SMTP connection). Locking down SMTP connections will help minimize the risk of spam from being sent from your network.
  2. Limit the number of emails that can be sent in a single day on a per-account basis. I used to leave this as ‘unlimited’ since I had a level of trust with my users that they wouldn’t be sending spam (I ran a very local ISP so I had met most of my customers face-to-face…which helps with trust). The problem I always ran into was that if someone’s computer got infected with a virus, it would send out a TON of spam or virus emails to propagate itself. If you set a limit relatively low (<250/day), it will be fine for most residential users. We’ve always had a a policy of increasing the limit to a much higher number if asked, but this helps eliminate problems due to our less savvy users getting their PCs infected. Note: our mail server software (Merak/Icewarp) allows for this setting to be controller per-user; some software may only allow for system-wide settings. In that case, you may have to start at a higher threshold if you have any business customers that use your mail server for commercial (but legitimate/non-spam) uses. I set our threshold incredibly low (100 messages per day) and only had to increase it for about 25 users (out of 3500).
  3. If you use BIND for your public DNS, DO turn off caching for all public IPs (that is: only answer queries for zones you’re authoritative for to “anonymous” requests but you can still answer recursive requests for “your” IP blocks).
  4. Enable SMTP authentication and allow an alternate port for access (due to what I mentioned in item #1 of this list). While you may primarily control SMTP relaying by allowing a certain range of IPs to send anonymously, you’ll inevitably have some users with a need to send emails while off-network.
  5. For your mail server: properly configure Reverse-DNS so that it matches the SMTP banner on your MTA and make sure there’s a matching Forward-DNS record for the public hostname/IP combination.
  6. Setup proper SPF records in your DNS. It only takes a few minutes and will greatly increase deliverability of your outbound mail.
  7. If you put your SMTP server behind a NAT’ed firewall, make sure that all outbound connections originate from the same public IP as inbound connections to it are on. In other words: don’t port-forward the SMTP port on a secondary IP of your firewall but allow the outbound connections on your mail server to just go out through the ‘default’ IP of your firewall. This will cause you a lot of grief when sending email to other organizations. It will also break your SPF configuration. Hint: search for 1-to-1 NAT in your router/firewall documentation.
  8. If you plan to host any SSL sites, place your DNS service on separate machines (i.e. don’t combine DNS and WWW services on the same box). I made the mistake of doing this and have had to do a bunch of firewall tricks to be PCI Compliant. In hindsight, putting DNS and WWW services on separate servers would have saved many hours of fiddling with settings on BIND and my firewall.
  9. When you setup a backup scheme for your databases (probably MySQL), set them up in a fashion so each database is stored in a separate dump file. Sifting through a >1GB mysqldump file to copy+paste the section that contains the tiny little database that your customer accidentally f’d up is no fun. Notepad doesn’t like it either.
  10. If your webmail interface uses SSL, spend the extra money and get a SSL certificate that matches the domain so your users aren’t prompted with a SSL certificate warning. Recent versions of IE and Firefox show the equivalent of a “doomsday warning” and will scare your users from using your webmail. I’ve had users call me to let me know that our server got hacked because they were so freaked out by the warnings. Thanks IE…..
  11. If your end-users are dial-up users, try like hell to implement per-message size limits that are less than 1MB…or steer your users to webmail if possible. I’ve been on more phone calls than I can remember, just trying to explain to ol’ Mrs Jones why she can’t download the 75MB high-resolution pictures of her grandchildren via POP3 on her 33.6k dial-up connection. Some people will understand that it’s just not possible to download these attachments over dialup….other people will blame you for disconnecting them from “the server” (Outlook and most other mail clients will time-out after 60 seconds of not receiving a complete message and show a generic “the server has disconnected” message). If you can steer them to webmail, it’s usually easier to view/download the messages since attachments can be downloaded 1 at a time…whereas POP3 has to get the entire message and ALL of the attachments as one single file.
  12. Stay away from mail servers that use the MBOX file format (or other single-file per-user data schemes). It may not be the case with all mail servers but I always ran into issues where the file would stay locked if a users session timed-out while opening/downloading a message….I always ended up having to kill the POP/IMAP process to unlock the MBOX file.

My ISP was/is primarily a Linux shop. I used the following when building the infrastructure:

  • OS: Debian Linux – http://debian.org
  • RADIUS (Authentication): FreeRADIUS – http://freeradius.org
  • DNS: BIND – http://www.isc.org/software/bind
  • Firewall: PFSense when NAT is used, IPTables when firewalling a Linux host
  • SSH/VNC/Remote Desktop client: VisionApp Remote Desktop – This software is awesome; you can flip between SSH, RDP, VNC and Telnet sessions using tabs…just like switching between web-pages using tabs in Firefox.
  • For servers that are low utilization (e.g. our Windows web server….I hated to set one up but some people swear by Frontpage): VMWare Server – http://www.vmware.com/products/server/
  • Linux/LAMP Server: We compiled MySQL+Apache+PHP from source but it was a total pain. Since the ISP I’ve grown to love DirectAdmin (http://directadmin.com). You can still compile custom options into PHP if necessary but it gives your end-users/customers their own WebUI to manage aspects of their site themselves.
  • FTP Server: if you decide to roll your own LAMP server (vs using something like DirectAdmin), I’d suggest VSFTP. It’s got everything you need for security built-in and it just works…probably the most maintenance-free service I manage.
  • Web Server: Hands-down, Apache. Again, I would suggest just using DirectAdmin. It’s pretty easy to compile from source though (if you don’t need/want a control panel and are comfortable editing config files to manage sites).
  • E-mail: Merak Mail Server, now known as IceWarp. I’d have to say I’m not a fan of this software anymore. It worked well for what we needed: it was stable, easy to administer, offers a good API for integration with our CRM and has decent webmail. my problem with it is/was that the spam filtering is somewhat of a black-box. I was never able to get decent support from the publisher of the software which meant that I had to find my own workarounds for dealing with spam that was “black holed” by the spam filters, even when the sender was whitelisted or the recipient had filtering turned completely off. In hindsight, I would recommend AtMail (http://atmail.com) – I’ve used it for other projects and it’s awesome (and Linux based!).
  • Remote Support: GotoAssist Express – there’s a monthly fee, but it’s well worth it when you have to do remote support. I’ve also used R-Hub’s Turbomeeting appliance which is equally as awesome and a one-time purchase instead of a subscription.
  • Webmail: Over the years, we went through Squirrelmail, DWMail, Roundcube, NeoMail, and eventually ended up just using the Icewarp interface since it was bundled with Merak and integrated the spam filters into each users’ webmail account. Of all of these, I would prefer/recommend Roundcube because it provides drag-and-drop and a fast-ish AJAX interface. Or AtMail :)
Share on Facebook