자동화된 인프라스트럭쳐 구축 2009.11. sysadmin 미니 세미나 발표 : 문태준 자동화된 인프라스트럭쳐 구축 2009.11. sysadmin 미니 세미나 발표 : 문태준 http://groups.google.com/group/sysadminstudy http://tunelinux.pe.kr 13 years as a Systems Administrator From garages to public companies How many people are systems administators? How many people are software developers? How many people conisder themsleves primarily concerned with business?

자료소개 이 자료는 Building an Automated Infrastructure 자료 및 "Web Ops 2.0: Achieving Fully Automated Provisioning" 를 기반으로 일부 수정한 자료임 해당 자료는 마지막 부분 참고자료에서 확인 Overview Why it’s important Talk about how to do it, and why it’s good, with monkeynews review Q & A

개요 시스템운영 – 현재의 상황 “자동화된 인프라스트럭쳐”란? 자동화 단계 질문과 답 Overview Why it’s important Talk about how to do it, and why it’s good, with monkeynews review Q & A

시스템운영 - 현재의 상황 수작업 (OS 설치, OS 설정, 애플리케이션 배포 및 설정) 서버에 직접 접속하여 시스템 관리작업을 함 시스템관리에 대한 프로세스, 정책이 부족 능숙한 SM이 필요하며 하위 관리자에게 위임하기 어려움

시스템운영 - 현재의 상황

자동화된 인프라스트럭쳐 구축 지정한 명세서에 따라 bare-metal 에서 비즈니스 서비스를 바로 실행할 수 있도록 전체 환경을 자동으로 전개함 개별 서버에 직접 접속해서 작업하지 않음 언제라도 특정 시점으로 복귀할 수 있음

자동화된 인프라스트럭쳐 구축 문제된 것을 고치는 것보다 다시 전개하는 것이 쉬움 지식이 적은 사람이라도 환경을 전개하거나 업데이트하는 것이 가능함 정책 및 그룹 기반의 시스템 관리를 수행함

자동화 단계 These are the steps to go from the napkin to actual deployment. OS Install - Get an operating system up and on a network DNS - Give your new system a name Server Inventory - Have a place where you keep track of each system, and what it does Identity Management - Grant your users access and privileges to your new servers Version Control - Keep track of the changes to your application code, and ideally, your infrastructure too Configuration Management - Keep track of how each system is configured, and update it when you make changes Monitoring - Watch your new systems for signs of trouble Trending - Make graphs and charts of important metrics, so that you can tell if the infrastructure is behaving well, and predict future capacity Application Deployment - Actually put your application on the infrastructure, and update it

자동화 단계 – OS 설치 수작업 각 시스템 수동설치 자동화 자동화된 설치 시스템 이용 (PXE) kickstart, Jumpstart, Windows RIS/WDS(Linux 도 설치 가능함) 등 표준화, 인증된 하드웨어 사용 및 절차 필요 Show of hands - how many people in this room can refer to every server in their infrastructure by name? How many people only have to go to one place to update the list of what hosts you have, and their IP addresses? The graph tells the story, I think. The time it takes to install and configure DNS is negligible, even if you have never touched it before. It’s worth your time, even if you only think about keeping everything in sync. Lots of good DNS tools, djbdns, bind, maradns. If you hate DNS, that’s fine - you can remove it entirely as long as you have configuraton management in place to update your /etc/hosts files or equivilant. The point is: have one place, centrally managed, that is canonical for the names of your severs.

자동화 단계 - DNS 수작업 각 시스템의 /etc/hosts 업데이트 자동화 DNS 서버 설치 설정 관리 Show of hands - how many people in this room can refer to every server in their infrastructure by name? How many people only have to go to one place to update the list of what hosts you have, and their IP addresses? The graph tells the story, I think. The time it takes to install and configure DNS is negligible, even if you have never touched it before. It’s worth your time, even if you only think about keeping everything in sync. Lots of good DNS tools, djbdns, bind, maradns. If you hate DNS, that’s fine - you can remove it entirely as long as you have configuraton management in place to update your /etc/hosts files or equivilant. The point is: have one place, centrally managed, that is canonical for the names of your severs.

자동화 단계 – 서버 인벤토리 수작업 엑셀파일, 위키 자동화 iClassify LDAP ControlTier 직접 만들기 Now that you have servers up, and they have names everyone can see, you need to keep track of the servers you have, and what they do. This may seem obvious, but I bet 90% of the startups I encounter, and 80% of the large companies, can’t tell you even *how many* servers they have with any degree of reliabaility, much less what each one is doing at any given time. (Even if they have DNS!) iClassify is a tool we created for doing just this job. It is a small agent that runs on each system, and repors to a centralized web service about the system it’s running on. You can then tag hosts, del.ici.ous style, and search the inventory with a full text search engine (Solr, for the curious.) I’ll talk more about it later. Also, Trusera, a client of ours, graciously let us use their actual infrastructure for these screenshots. Thanks, Trusera. :) LDAP often already exists for Identity Management in many infrastructures, and as long as you don’t need a lot of complex data, it’s a good place to put your host information. Lots of people have written databases that do this sort of thing. Use whatever suits you -- but I have to say, making the systems report themselves to the inventory system is a huge, huge win.

자동화 단계 - 계정관리 수작업 각 계정 수동 추가 자동화 LDAP 또는 AD 이용 설정 관리 All that, and we still don’t have users everywhere yet. MonkeyNews is a small company right now, only two people, and six servers. But you still have to figure out who has access to which servers, and what privileges they have. The 수작업 way to do this is to add each user on every system. The 자동화 way is to use a centralized service, such as LDAP or Active Directory. This graph should look familiar, because it has the exact same automation bonus as DNS does. When you have 5 servers, the 5 minutes it takes seems like no big deal. But that’s 5 minutes for *any user change*. Password change? 5 minutes. And the curve is linear.. As you add more servers, you have to add users everywhere, and it takes longer and longer. Centralize your identity management infrastructure. Have one user name and password.

자동화 단계 - 버전관리 수작업 파일 복사, FTP, http 이용 자동화 버전관리 툴 이용 CVS, Subversion, 기타 개발뿐만 아니라 운영에서도 버전관리 필수 Having a central place to track changes to code and infrastructure, with blame and history Not really an “자동화” vs “수작업” thing - you just don’t have a choice :) Using version control is a requirement of at least two future steps Subversion, Git, Mercurial, CVS Perforce Just pick one you like and use it religiously

자동화 단계 - 설정관리 수작업 버전 관리 시스템 위키 SSH 루프 이용 설정 파일을 복사한 후 체크 작업한 내용을 기록해야 함 Server Classification says what a thing ought to be, Configuration Management makes it so. Everything up to deploying your application specific code on all of your servers This means everything that isn’t done for you at OS installation

자동화 단계 - 설정관리 자동화 Cfengine Puppet Bcfg2 Slack 코드로 인프라스트럭쳐를 관리하자!!! 자동화 configuration management is the heart of having an 자동화 infrastructure Instead of doing things by hand and keeping track of them You express how the infrastructure should behave as code Cfengine is the grand old academic dean of Unix/Linux configuration management Puppet is, in my opinion, the current state of the art Bcfg2 I have never used, but some folks dig it - XML based config files Vertebra a new entry here? Let me show you what I mean with a puppet example that everyone can relate to, managing /etc/sudoers

자동화 단계 - 설정관리 자동화가 핵심 50+ 이상의 서로 다른 애플리케이션 왜 반복작업을 하냐? ad, apache2, apt-proxy, beaver, build-essential, capistrano, yum, apt, djbdns, emacs, erubis, rubygems, iclassify, imagemagick, iptables, java, logrotate, man, maradns, memcached, mongrel-runit, munin, mysql, nagios, nscd, ntp, openldap, openssh, perl, perlbal, php, postfix, postgresql, puppet, rails, resolver, rsync, ruby, runit, sqlite, subversion, sudo, trac, zsh 왜 반복작업을 하냐? Easy to adapt to wide variation between systems Incredible time savings Always current

자동화 단계 - 모니터링 수작업 각 서버 수동 추가 자동화 시스템 인벤토리, 설정 관리 툴 이용 Monitoring, for our purposes, is the act of watching the system for conditions that we want to be notified about. Things like “is this service running”, or “did I make enough money in the last hour”. In a 수작업 world, you would configure each server (and service) to be monitored by hand In an 자동화 one, you would configure each class of server one time, and let the automation do the rest Only edit the config files once for each kind of system Tools like Nagios can be 자동화 with configruation management, tools like Hyperic and OpenNMS have there own discovery mechanisms

자동화 단계 – 리소스 모니터링 수작업 각 서버 수동 추가 자동화 시스템 인벤토리, 설정 관리 툴 이용 “The process of extrapolating metrics to make future capacity forecasts.” Charts and Graphs Has a similar configuration burden as Monitoring, and an identical solution

자동화 단계 - 이메일 모든 서버는 이메일 발송이 가능해야 한다. Modern web applications send a ton of email, so make it easy to do Most linux distributions will send lots of email, so make it easy on them too Monitoring will want to send email Use your configuration management system and system inventory to automate the configuration on a per-server basis Or do it by hand

자동화 단계 – 애플리케이션 배포 수작업 수작업으로 애플리케이션 설치 자동화 Capistrano ControlTier 시스템 인벤토리 툴에 통합 Application Deployment can be easy, or it can be hard It can be time consuming, or it can be very quick The key predictor of either is the number of steps you have to take to deploy If the number is 0%, and a deploy either succeeds or fails, you have a 0% chance of a deploy related production impacting outage As the number of steps increases, so does your odds of screwing it up Someday, Vertebra!

다시 확인 자동화된 인프라스트럭쳐의 의미 : 자동화된 인프라스트럭쳐 “사람의 개입 없이 대규모 사업을 운영하는데 필요한 기본 서비스를 갖추는 것” “고양이가 사용할 수 있는 시스템” 자동화된 인프라스트럭쳐 시간 절약 효율성 늘림 확장성 가능함 경제적인 충격을 줄임 유연성 개선

다시 확인 자동화된 인프라스트럭쳐의 의미 : 자동화된 인프라스트럭쳐 “사람의 개입 없이 대규모 사업을 운영하는데 필요한 기본 서비스를 갖추는 것” “고양이가 사용할 수 있는 시스템” 자동화된 인프라스트럭쳐 시간 절약 효율성 늘림 확장성 가능함 경제적인 충격을 줄임 유연성 개선

참고자료 Building an Automated Infrastructure (O’REILY Velocity 2008) : http://en.oreilly.com/velocity2008/public/schedule/detail/2238 Achieving Fully Automated Provisioning : http://www.dtosolutions.com/storage/downloads/FullyAutomatedProvisioning_Whitepaper.pdf Reliable, Repeatable, Reproducible Infrastructure http://sysadmin.miniconf.org/presentations08.html#02 ControlTier http://wiki.controltier.org/wiki/ControlTier Continuous integration http://en.wikipedia.org/wiki/Continuous_integration Automating Linux and Unix System Administration Second Edition The Practice of System and Network Administration (2/E) http://groups.google.com/group/sysadminstudy http://tunelinux.pe.kr/

질문과 답