Arquivo para categoria Projetos

Profiling your web application with wsgid request filters

It’s always important to know how your web app is performing and there are many ways to do this. One such way is measuring memory consumption, for example. This blog post will explore how you can measure response time on your live application.

We will see how to make this using a new feature available in wsgid 0.7.0:  RequestFilters.

Request Filters

This feature works in a very simple way. Wsgid gives you the opportunity to run external code inside the regular request/response flow. Wsgid defines two distinct interfaces: IPreRequestFilter and IPostRequestFilter. By implementing any of these interfaces you will have your code called on every request.

Since we are injecting external code into the execution flow of our application, wsgid must ensure that a failing filter does not crash a successful request. So unless you call sys.exit(0) or crash the python interpreter (believe me, it happens!) all requests should complete as usual.

IPreRequestFilter Interface

This interface has just one method (code here):

def IPreRequestFilter(plugnplay.Interface):

    def process(self, m2message, environ):
        pass

The m2message parameter is an instance of wsgid.core.Message. This is the parsed mongrel2 message. The environ is the WSGI Environ, as described by the PEP-333 specification.

Your code can modify the environ freely. This modified environ will be passed to the running WSGI app.

IPostRequestFilter Interface

This interface has two methods (code here):

def IPostRequestFilter(plugnplay.Interface):

    def process(self, m2message, status, headers, body):
        pass

    def exception(self, m2message, e):
        pass

Since this filter is called after the WSGI app has ran, the filter receives all values returned by it. These values are are the raw values as defined by PEP-333. The exception method is called when the WSGI app call fails for any reason (usually an unhandled exception). From the wsgid perspective, even if the application returns an HTTP 500 it will be considered a successful run. If you want to detect error HTTP responses you should inspect the status parameter, received by the process method.

Important note: If the WSGI app call fails, just the exception method will be called. Does not make sense to call the process method since the WSGI app did not return any value.

Simple examples

The wsgid source-code includes two simple examples of request filters. The first is the WorkerPidFilter. This just adds an header to the response containing the PID of the Wsgid worker that ran the WSGI app. The code is very simple:

class WorkerPidFilter(Plugin):
 '''
 Simple fillter that adds one more response header containing the
 pid of the Wsgid workers that was running the WSGI application
 '''
    implements = [IPostRequestFilter, ]

    def process(self, message, status, headers, body):
        return (status, headers + [('X-Worker', os.getpid())], body)

    def exception(self, message, e):
        pass

Another example is a filter that calculates the total elapsed time of your request. You can take a look at the code here.

A more real world example

These example filters works very well for simple cases. When it comes to real world production application we may have to do it a little different. First of all, when writing a request filter we must consider the time the filter will take to run. The less it takes, the less it interferes in the request response time.

We could, for example, save the response times in a database. But we know that this wouldn’t be a great idea in the end! So the idea here is to save the information to later processing. We could save it into the logs. You can do this by just importing wsgid.core.log. This is a stantard Python Logger and it writes the information into your app main log file (the file is located inside your wsgidapp folder at logs/wsgid.log). But by doing this we would have to keep parsing the application log, not a very good idea either.

Another idea is to use a queue server, such as RabbitMQ. This way we can keep the filter execution time very low and we will have all messages saved to be processes later, without the need to parse any log file.

Attached to the RabbitMQ server we would have a separated process, reading all the messages and storing them anywhere, in a database, in a Graphite server or any other place.

The idea behind this implementation

The idea is very simples. Since your filter receives the parsed mongrel2 message, you have access to the connection ID of the client making the current request. This could be your primary key when measuring the values. In your PreRequestFilter you can send to the queue server, along with the connection ID, all the HTTP headers, the current time and any other information you want.

When the PostRequestFilter runs, you send the connection ID, the status returned by the WSGI app, the response headers and the current time again. With this information you can not only calculate the response time (per request URI, if you like) but you can calculate all sorts of information about your requests. User-Agent percentage, Content-Length average size, the most/less visited URI, HTTP status distribution (how many 200’s, 400’s, 500’s is your WSGI app returning) just to cite a few.

I will leave this implementation for a future post as I pretend to calculate all this information for the wsgid (http://wsgid.com) website and for my personal website (http://daltonmatos.com). This will be perfectly possible since they both run with wsgid.

Thanks for the reading and I hope you enjoyed!

Anúncios

, ,

Deixe um comentário

How to configure wsgid and mongrel2 to handle arbitrary sized requests

Intro

Wsgid is a generic handler for the mongrel2 web server. It acts as a bridge between WSGI and mongrel2 and makes it possible to run any WSGI python application using mongrel2 as the front end. To know more about both projects visit their official websites at: http://wsgid.com and http://mongrel2.org

In this post we will see how to configure mongrel2 and wsgid so your application will be able to handle arbitrary sized requests very easily and with a low memory usage.

Why?

The problem of receiving big requests is that depending on how your front end server deals with it, you can easily become out of resources and in a worst case scenario your application can stop responding for a while. If your application exposes any POST endpoint, then you should be prepared to handle some big requests along the way.

The big picture

Mongrel2 has a very clever way to handle big requests without consuming all your server’s resources, in fact without consuming almost nothing, only bandwidth. Basically what it does is to dump the entire request to disk, using almost none additional memory. Here is how it works:

When the request comes in, mongrel2 sends a special message to the back-end handler containing only the original headers. This message notifies the start of this request. It’s up to the handler to accept or deny it. To accept the request, your handler just do nothing. If you want to deny the request, your handler must send a 0-length message back to mongrel2. Due to mongrel2 async model, you can send this deny message at any time, does not need to be imediatelly after receiving the request start message.

If you happen to be using wsgid (I hope you are!) this happens without your WSGI application ever knowing. All your application will see (if at all) is the wsgi.input attribute of the WSGI environ (if you happen to write a WSGI framework) or just request.POST (or something similar depending on what framework you’re using). It’s all transparent.

After the request is completely dumped, mongrel2 sends another message notifying the end of the upload and that’s when wsgid will actually call your application. During all the upload process, your application does not even know that a new request has come and was being handled.

All this happens while mongrel2 is already dumping the request content to disk. If your handler happen to deny the request, mongrel2 will close the connection and remove the temporary file. To know more about how mongrel2 notifies the handlers, you can read the on-line manual: http://mongrel2.org/static/mongrel2-manual.html.

Configuring mongrel2 to handle big requests

The configuration of the server is extremely simple. All you have to do is tell it where to dump any big request that may arrive. You may ask: How big a request must be to be dumped to disk? The truth is that you decide this and tell mongrel2 about it.

To tell it where to dump the request you must set the option upload.temp_store to the correct path. This path must include the filename template. An example could be: /uploads/m2.XXXXXX. This must be a mkstemp(3) compatible template. See the manpage for more details.

The size of the request is set with the limits.content_length. This sets the biggest request mongrel2 will handle without dumping to disk. This size is set in bytes.

Configuring wsgid to understand what mongrel2 is saying

Since mongrel is saving requests on disk wsgid must be able to open these files and pass its contents to the application being run. It’s important to note that the path chosen on the upload.temp_store is always relative to where your mongrel2 is chrooted, so somehow we must tell wsgid where this is.

Fortunately this new release of wsgid comes with a new command line option: --mongrel2-chroot. You just have to pass this to your wsgid instance to be able to handle big requests.

Alternatively you can add this options to your wsgid.json configuration file if you want. This can be complished with a simple command:

$ wsgid config --app-path=/path/to/your/wsgid-app --mongrel2-chroot=/path/to/mongrel2/chroot

This will save this new option on your config file. Just restart your wsgid instance and it will re-read the config file.

Working without knowing where mongrel2 is chrooted

There is another way to handle these big requests. In this approach your don’t need to pass --mongrel2-chroot to all your wsgid instances. The trick here is to mount the same device on many different places. The easiest way to do this is to have mongrel2’s temp_store in a separated partition (or logical volume if you are using LVM). Let’s see an example:

Suppose we have mongrel2 chrooted at /var/mongrel2/ and configured this way:

upload.temp_store = '/uploads/m2.XXXXXX'

Since mongrel2 assumes /uploads is relative to its chroot we must mount this device ate the right place.

# mount /dev/vg01/uploads /var/mongrel2/uploads

Now our logical volume is mounted at /var/mongrel2/uploads. There is one last setp so we can start serving big requests with wsgid. When mongrel2 sends the upload started message to wsgid, which contains the path of the temporary file, the path received by wsgid is the same we put on mongrel2’s config. So wsgid will try to read /uploads/m2.384Tdg (for example) and obviously will fail. So we need a way to make /uploas/ also available to wsgid (that normally is not chrooted anywhere) and the trick to do this is to re-mount the same device at a different place. And this is how we do it:

# mount /dev/vg01/uploads /uploads

So now we have the same device mongrel2 writes the temporary files available to all wsgid processes that need to read them. Remember that mongrel2 does not remove any of these file because obviously it does not know when your app is done. So it’s up to you to clean them.

If your /upload directory is part of a bigger volume or is not on a separated one, you still can accomplish this multi-mount approach. Just use the --bind. This way you can re-mount any directory at another place. Read the mount man page for more details.

Conclusion

This is another cool mongrel2 feature that the latest wsgid release (0.5.0) already supports. So now wsgid supports all major features mongrel2 provides and is starting to be more mature, trying to find its way into a production-ready tool.

This release complete changelog is available at wsgid.com/changelog and you can grab your wsgid copy at wsgid.com.

Thanks for reading and enjoy!

, , ,

Deixe um comentário

How I’m planning to test one of my projects by writing another one

The purpose

Some time ago, I started writing wsgid. It is a project that brings to you the ability to run any WSGI application with mongrel2 web server. Some time ago I had this idea of writing a web application that could help me test wsgid on a real production environment. Both wsgid’ official website and my personal website are hosted using wsgid and mongrel2 as the backend, but none of them even uses a database (and maybe won’t for a long time) and since they are very simple and low traffic websites, I think they aren’t good enough to test wsgid.

I thought about some type of applications that would be better to test wsgid and end up with the decision to write my own blog engine. This same engine will soon power my own blog, self hosted and managed by me.

Wait, another blog engine? Why?

Yes, it will be a blog engine, another one! =) Before you think about NIH, I will try to explain why I decided to start this project. The very first thing I did when I thought about writing one more blog engine was to find some existing ones. But it could not be any blog engine, they must obey some rules:

  • Written in python
  • Used a WSGI framework
  • Bonus points for engines written in Django

These rules are obvious. If I’m trying to test wsgid (that is a WSGI gateway) the application must be written in Python and be compliant to the WSGI specification. The last one was a bonus point because I’m currently learning Django. I found that are there plenty of Django blog engines, but one decision made me not choose any of them: I want to learn something new, and I think the best way to do this is having a project. And not any one, but one that you will really use. And writing the code of my own blog is a good way to maintain the project alive, evolving and getting better and better.

Another point that helped me on this decision was the willing to self host my applications. This will give me very important knowledge and experience on system administration, servers, deployment, clusters and may more. Recently I had two amazing opportunities to work with these topics and I’m sure that if I were already managing my own infra-structure for a while I could have done much better on these two interviews.

Until the day of witting this blog post, my blog is still hosted on wordpress.com. I  plan to migrate to my own servers as soon as this new project becomes minimally usable. This will be good for many reasons that I already said and one more is that I will be able to have my own domain name and not pay any more money for this, since I already own daltonmatos.com.

Wish me luck

Starting a new project is always a great responsibility, first with yourself, second with the people that follow your projects and more important: with who uses your project. So today I decided to start this new project: The blog engine. It has no name yet, the only certainty is that I will create this project and host my own blog. The code will be hosted on github, where I publish all my codes. So if you are intersted follow me there and stay tuned!

Thank you for reading!

, ,

Deixe um comentário

The importance to share your knowledge and your projects

In this post I will talk about how important is to me to have side projects. And how these same projects gave me some considerable visibility to other developers and also to big tech companies. I talk also about what I learned after being part of the hiring process of Google and Facebook, both opportunities thanks to my projects and my publishing.

Visibility

Sometimes you just want to learn something new, and one of the greatest ways to do this is having a project. You don’t need to have a project that is used by many people. That’s not what measures whether a project is good or not, a project is good when you care about it. More important, a project is really good when you learn something useful during its development.

Other than that, your project can be your showcase, so use it also to show others what you are capable of doing. There are other ways to show yourself to the world, and one of them is be part of the community around someone else’s project. But you can’t just choose a random project, the project must be important to you, either because you are interested or because you use it to solve an important problem. And remember, always give valuable feed backs, don’t just make noise. If you have nothing to say, so just don’t say it.

First, the Projects

I always try to convince who is near me that having side projects are a great idea! I’m always trying  to help my friends with their projects, sending patches, doing pair programming or whatever I can do to help. Until 1 year ago I didn’t have any side project, but not because I didn’t want to, it was just because none of them were such a big deal. And then when I started to study python I saw the first opportunity to have one project. At that time I was deploying a Trac instance and because of this I started to read its source-code. When I realized how amazing Trac’s internal component management was I decided to create my first project, and then plugnplay was born. This project gave me the knowledge about the python language and at this time I already had chosen python as my language of choice.

The idea for the second one came when I found out about another project named mongrel2. Actually I don’t remember how I came across mongrel2, but as soon as I realized that it lacked python support, added the fact that I had just chosen python as my language, I decided to start wsgid. Wsgid provides WSGI support for mongrel2. As I wanted this project to be more presentable I also bought a domain name (wsgid.com) and made an official website (that runs using wsgid itself!). Over time, wsgid earned its place on mongrel2 official site, where it shows all supported languages. This was, with no doubt, a big move for the project and gave me even more motivation to move further.

The first contact, Google

So there I was, developing my projects as always when suddenly I see in my inbox an email from a @google.com address. At that exact moment I thought: “Ok, Gmail spam filter has failed. Let me mark this and go on with my life.” But then I decided to just do a little research about that message. I looked at the original communication between the smtp servers, and all that seemed real. The email talked about my contributions to the open source community and my blog, and the recruiter was asking if I was interested in a full-time opportunity at Google.

So I decided to reply to that message and two things could happen. Or I would be eternally without response, and then I would confirm that it was spam. Or I would receive another response and realize that is was real! And that’s what happened. I was absolutely happy and proud to know that Google was offering me the opportunity to be part of its hiring process.

This was January this year, and then I started chatting with the recruiter. I had two phone calls. The first was not technical, was just to know a bit more about me, my background, my time in college, my current work and some other things. After this we scheduled the first technical interview. I, obviously, was very nervous before and during the interviews, mainly because all them was made in English, that is not my mother language.

The second phone call (first technical) was ~20min chat with general questions about system administration, network, etc. I tried my best, considering the poor call audio quality. I had to ask the interviewer to repeat several times, some of them he didn’t. Some weeks later I received another call telling me that we were not moving forward with the process, and so my chances with Google were finished.

The second, Facebook

Some weeks ago, around end August, it happened again. And this time was from a Facebook recruiter. Same story, and this time they saw my blog and since one of my projects was server related (wsgid), they contacted me about a full-time sysops position. This time the domain of the sender was unusual to me, it was @fb.com. But a simple DNS query was able to show me that fb.com was in fact controlled by facebook. This made me feel more confident about all that story.

And, again, I decided to accept the challenge. So I started chatting with the recruiter and on the next week I was already with a scheduled phone interview. This first interview was an elimitaory one, just to know if they would continue with the process or stop right there. Again, the audio quality of the call was awful, but different from Google interview, every time I asked the person to repeat the question I was attended. This first call was very quick (~20min) and some of the questions were about protocols, command line tools and system administration in general. I answered right almost all the questions, and then the interviewer finished the call with a compliment. Saying that I was pretty good at Linux. =)

The next week was time for another phone call, this time a non-technical one. The recruiter just wanted to give me valuable tips about the overall process and specifically about the interview that was to come, the first really technical one. And the day has come, I already knew that the interview would be a ~45min call with 2 or 3 problems to be solved. They used collabedit.com to see the code I was writing.

During the one week I had to prepare myself, I studied algorithms, data structures, grabbed some problems from programming competitions and solved them. As I was focused on algorithms, during the interview I got stuck trying to solve the problem with an algorithm written in python. The position was to sysops, that is, heavy linux server users with great command line powers. Even though I simply didn’t try to use bash during the call. After almost 20 min trying to solve this with python code, the call was finished by one of the interviewers and he said that soon they would contact me again. As soon as I hang up the phone, a one-line bash solution came into my mind!

What I learned about all this?

Even not being selected on both opportunities, something good comes out of it. When all this ended and I started to think about it, I realized that my simple projects, started mainly to learn something new, went beyond all my expectations! All my projects are published on my personal website: daltonmatos.com. What all this taught me is that is all the effort put on my own projects, the contributions I made to other projects, all these worth it!

I also realized that big companies are always looking for good developers out there and one of these could be you, the same way these two times it was me.

So what can you do about this?

What I would like you to understand is that whenever you have some code that solves a specific problem for you, publish it! You can be sure that this your solution can be useful for many others. I already saw some friends not publishing their code because of many weird reasons, from “the code is still too bad, let me evolve it a bit” to “This is a super-secret project, somebody will end up stealing my super-coll idea”. If you also think like this, I hope that the story I just told you changes your mind, at least a little bit!

So go start your projects or publish an already written one!

, ,

2 Comentários

Finalmente meu site pessoal está lançado: daltonmatos.com

 

Hoje é um dia muito importante. Finalmente depois de muito tempo consegui lançar meu próprio site. Já tinha comprado o domínio há bastante tempo mas não tinha ainda parado pra escrever os textos e o código do site.

O site é uma aplicação django e roda com o auxílio de um projeo meu já citado aqui no blog, o wsgid. Por enquanto o blog continua aqui no wordpress.com mas pretendo migrá-lo pra lá, pois assim terei tudo centralizado. Ainda tenho que encontrar uma blog engine para que possa migrar o blog.

Bem, é isso aí! Vai lá e dá uma olhada: http://daltonmatos.com

, ,

2 Comentários

O que aprendi desenvolvendo projetos de código aberto

Nesse post vou escrever um pouco sobre minha primeira experiência no desenvolvimento de projetos de código aberto.

O interesse por software de código aberto

Desde muito tempo tive interesse no desenvolvimento de software, especialmente em softwares de código aberto. Um interesse especial é o fato de você ter a oportunidade de ver códigos escritos por outras pessoas. Ler código de outras pessoas nos faz aprender muito, pois vemos abordagens diferentes para um mesmo problema e podemos aprender hacks que não fazíamos idéia que seriam possíveis.

Sempre publiquei meus códigos para quem quisesse ver e defendo a idéia de que as pessoas deveriam também sempre publicar seus códigos. A unica situação em que faz sentido não publicar algum código é quando esse código é crucial para um negócio qualquer. Nesse caso não faz sentido entregar o ouro, claro.

Primeiras contribuições

O primeiro passo para ter seu projeto de código aberto é fazer parte de algum outro projeto. Fazendo isso você pode ter contato com outros desenvolvedores, aprender como funciona o projeto especificamente, como é o processo para que você possa contribuir, além de ganhar experiência para que um dia possa tocar seu próprio projeto de código aberto.

Minha participação em projetos de código aberto teve início quando comecei a estudar python, há pouco mais de um ano atrás.  Antes de escrever código em python, lia bastante sobre a linguagem. Aprendi lendo o próprio site da linguagem e sempre que tinha alguma dúvida específica sobre alguma sintaxe, ia direto ler a gramática. Acho essa forma mais eficiente do que ler hello worlds por aí.

Em uma dessas pesquisas, estava tentando encontrar um framework web que fosse bem simples de entender, afinal eu estava começando a estudar a linguagem. Apesar de já ter experiência com sistemas web, conhecer sobre HTTP, requisições e etc. Web + python era, naquela época, algo novo.

Foi então que encontrei um framework chamado pyroutes. Pois bem, desde o início gostei da implementação pois ela faz uso dos decorators, que é uma coisa que acho fantástica na linguagem. Então comecei a estudar o código desse projeto, fiz meu fork no github e comecei a enviar pull requests. Comecei simples, apenas corrigindo erros triviais nos testes automatizados: 2d921fa9122a3211. Pouco depois, já estava corrigindo bugs e fazendo implementações menos triviais: 44d1708.

O que ganhei com tudo isso? Bastante experiência! Contato com pessoas de outras nacionalidades, conhecimento na linguagem que estava aprendendo e o mais importante: A oportunidade de outras pessoas me conhecerem e saberem das minhas habilidades e conhecimentos. Nesse projeto especificamente ganhei outra coisa também muito impotante: permissão de commit no repositório oficial!

A hora de começar meu próprio projeto

Depois de ter colaborado por algum tempo com esse projeto decidi que tinha chegado a hora de tentar uma nova empreitada: Tocar meu próprio projeto de código aberto. A idéia desse projeto apareceu ainda quando estava estudando sobre a linguagem. Como já disse aqui, acho muito importante ler código de outras pessoas e foi lendo o código do Trac que tive a idéia para o meu primeiro projeto: plugnplay.

O desenvolvimento desse projeto foi bem legal, foi meu primeiro código publicado que realmente tinha chance de ser útil para alguém além de mim, e isso é fantástico. Foi desenvolvido em apenas uma semana (de acordo com o histórico de commits).

O projeto é bem pequeno e fiquei bem feliz quando vi que ele já tinha 3 watchers!

O Segundo projeto

É claro que não parei por aí! Continuei com minhas pesquisas até que surgiu a idéia para o segundo projeto: wsgid. Essa idéia surgiu quando me deparei com um outro projeto: mongrel2. O objetivo do wsgid é me ajudar a ter uma infra-estrutura totalmente automatizada para fazer o deploy de minhas aplicações, e esse é o primeiro passo para que consiga chegar a esse objetivo.

Ao contráio do plugnplay, para esse projeto fiz um website, uma documentação completa e até comprei o domínio (wsgid.com), ou seja, coisa séria! =) E isso se mostrou muito importante. Querendo ou não, um projeto que apresenta um site oficial e uma documentação completa de suas funcionalidades atrai muito mais a atenção das pessoas, pois passa uma credibilidade maior. Uma coisa interessante é que o site oficial do wsgid roda usando o próprio wsgid!

A questão da documentação é especialmente importante pois temos que lembrar que nem todo mundo (mesmo desenvolvedores) terá curiosidade suficiente para ler o código-fonte para poder aprender a usar seu projeto, então ter uma forma fácil de começar a usar o projeto se mostra muito importante.

Visibilidade do projeto

O projeto wsgid teve uma visibilidade muito maior do que o plugnplay, e acho que isso se deve a alguns fatores. Em primeiro lugar ele é um projeto satélite de um projeto grande, o mongrel2. Só isso já te dá mais visibilidade. Outra coisa é que como eu estava participando da lista de desenvolvimento do mongrel2, mandei emails para essa lista divulgando o projeto e recebi feedbacks importantes. O fato da URL do site oficial estar na assinatura de todos os meus emails também ajuda. Novamente, isso só é possível pois o projeto possui um website.

Durante o desenvolvimento do projeto (até hoje, na verdade) ainda recebo notificações por email quando o projeto ganha mais um watcher em seu repositório. Hoje o projeto está com 13 watchers, o que é fantástico! Esse tipo de coisa te dá mais gás para continuar escrevendo o código, pois você vê que existem pessoas que acharam seu projeto interessante a ponto de decidirem acompanhar sua evolução. E isso é muito bom! Outra coisa ainda melhor é o número de forks que o projeto recebeu, até agora já são 5 em 6 meses de projeto.

Novamente, durante o desenvolvimento desse projeto tive a oportunidade de me comunicar com outras pessoas. Como já disse, entrei para a lista de desenvolveores do projeto mongrel2 e novamente comecei a submeter patches com pequenas correções. Depois, quando já estava com um domínio maior sobre o código já comecei a enviar códigos menos triviais, participei de várias discussões na lista até que, novamente, ganhei permissões de commit no repositório oficial. Essa é a lista de commits que fiz no repositório oficial, até o momento.

Conclusão

Minha conclusão disso tudo é que é muito divertido contribuir com algum projeto, melhor ainda é ter seus próprios e receber contribuições de outras pessoas  (já recebi um patch para o wsgid). Ter um side-project te dá mais conhecimento, faz com que as pessoas possam te conhecer (e isso é muito importante!) e te permite aprimorar suas habilidades.

Se você ainda não posui um projeto, comece um! Os dois que tenho surgiram para resolver problemas que eu tinha no dia-a-dia, tenho certeza que você também já viveu situações onde um programa poderia te ajudar ou até mesmo resolver seu problema. Pense que outras pessoas certamente já tiveram o mesmo problema e também poderiam ser ajudadas pelo seu projeto.

2 Comentários

Rodando sua aplicação WSGI como um *nix daemon

For english readers, pease see post below.

Quase sempre quando pensamos em aplicações web (eu inclusive), pensamos na apache stack. Esse post apresenta uma forma diferente de fazer o deploy de sua aplicação, começando pelo fato de usarmos o servidor web apache.

Primeiro de tudo, precisamos conhecer uma peça chave diso tudo:

Mongrel2

Mongrel2 é um servidor web escrito por Zed A. Shaw. Esse é um servidor que simplismente não se importa em que linguagem sua aplicação foi escrita, tudo que você precisa fazer é seguir algumas regras bem simples para conseguir juntar mongrel2 e sua aplicação.

A primeira regra é que sua aplicação precisa saber se comunicar através de uma fila, nesse caso ØMQ (http://zeromq.org) e a segunda regra é que sua aplicação precisa entender o protocolo definido pelo mongrel2. Você pode (e deve!) saber mais sobre o mongrel2 no site oficial: http://mongrel2.org.

A grande sacada do mongrel2 é que, o fato dele usar uma fila para se comunicar com sua aplicação faz com que sua aplicação possa rodar de forma desacoplada do servidor, isso significa que sua aplicação não roda com a ajuda de um mod_algumacoisa ou como um processo filho do servidor web. Uma outra vantagem é que o ØMQ permite enviar mensagem por TCP, isso significa que sua aplicação pode estar rodando em uma máquina diferente de onde o mongrel2 está rodando.

Poder rodar sua aplicação em máquinas diferentes é fantástico pois sempre que necessário você pode adicionar mais uma máquina ao seu cluster e startar mais instâncias da sua apliação, todas as instâncias vão se comunicar com o mesmo mongrel2, que vai balancear as requisições entre todas as instâncias conectadas, usando uma politica round-robin.

wsgid

wsgid é o projeto que desenvolvi para tornar possível rodar suas aplicações WSGI usando como servidor web o mongrel2. O wsgid é a ponte entre o mongrel2 e a especificação WSGI (PEP-333), isso significa que basta a sua aplicação obedecer à especificação WSGI para já estar pronta para rodar com o mongrel2.

O wsgid entende tanto a língua do mongrel2 (zeromq + protocolo) quanto a especificação WSGI. Além disso, sua aplicação agora é um processo separado, com PID e tudo mais. Isso te traz algumas vantagens, como:

  • Pode rodar com permissões de qualquer usuário, a sua escolha;
  • Por ser um processo do sistema operacional, está automaticamente sujeito a quaisquer configurações que o S.O possa oferecer, como por exemplo: limite de banda, memória, CPU e etc;
  • Pode ser rodado dentro de um chroot, caso você precise rodar código não confiável;
  • Entre outras.

Iniciar uma nova instância de sua aplicação é tão fácil quanto:

   $ wsgid --app-path=/path/to/wsgid-app-folder/ --recv=tcp://127.0.0.1:8888 --send=tcp://127.0.0.1:8889
      --workers=4

Nesse exemplo simples estamos conectando ao endpoint 0MQ que está na mesma máquna em que nossa aplicação, mas nada impede de usarmos –recv=tcp://129.168.0.2:8888, sendo esse IP o de uma máquina qualquer em seu cluster. Ainda nesse exemplo a opção –workers=4 cria 4 processos que vão ser responsáveis por atender requisições de sua aplicação.

wsgid, principais funcionalidades

Além de facilitar o deploy de aplicações WSGI com o mongrel2, o wsgid possui ainda outras funcionalidades importantes:

workers

Com a opção workers você pode iniciar quantos processos desejar, de uma só vez. Isso significa que você não precisa rodar o comando 4 vezes caso queira 4 instâncias, apesar de você poder fazer isso, sem problema nenhum. Mas essa opção te dá a possibilidade de fazer –workers=4, que te dará o mesmo resultado.

keep alive

Com a opção keep-alive o wsgid reinicia automaticamente qualquer processo que tenha terminado sua execução. Isso significa que uma chamada ao wsgid com –workers=4 –keep-alive manterá sempre 4 workers trabalhando para sua aplicação.

hot deploy

Sempre que o wsgid restarta um dos processos, o código de sua aplicação é recarregado, afinal é um novo processo que está sendo criado. Isso significa que você pode atualizar o código-fonte de sua aplicação e apenas mandar um SIGTERM para todos os seus workers, isso fará com que o wsgid restarte cada um deles, mas dessa vez já rodando o novo código-fonte.

chroot

O wsgid já pode, automaticamente, fazer o chroot para uma localidade onde a aplicação está. Isso faz com que essa aplicação possa rodar de forma isolada, criando assim um ambiente um pouco mais seguro (não totalmente) caso você esteja rodando códigos não confiáveis.

Sistema plugável para carregar diferentes aplicações

O wsgid conta com um sistema de plugins muito simples porém bem poderoso. Com ele você pode escrever seu próprio Application Loader, caso o wsgid não consiga carregar sua aplicação. Com isso será muito simples adicionar suporte a outros frameworks WSGI para que mais aplicações possam fazer uso desse projeto.

Espero que tenha gostado do projeto, qualquer feedback é bem vindo. Para saber mais sobre o projeto visite o site oficial: http://wsgid.com e saiba mais.

Mas lembre-se, use o que for melhor para você, use o que te atende melhor. Não estou dizendo que a solução mongrel2+wsgid é a melhor sempre, estou apenas dizendo que essa combinação é fantástica, e deve ser considerada no momento de publicar uma nova aplicação WSGI.

Apenas por curiosidade, o site wsgid.com é uma aplicação django e está rodado usando o próprio wsgid.


Running your WSGI app as a *nix daemon

Almost every time we think about web apps (at least me) we think about the apache stack. This post will show you a different method to deploy your aps, starting by not using apache as the web server.

First of all, we need to know a key piece of all this:

Mongrel2

Mongrel2 is a webserver written by Zed A. Shaw. The server is language agnostic, which means that it just doesn’t care which language you wrote you app, all you have to do is follow some very simple rules and you will be able to plug mongrel2 and your app together.

The first rule is that your app must know how to communicate through a queue, in this case ØMQ (http://zeromq.org) and the second one is that you must follow the very simple protocol specified by mongrel2. You can (and must!) know more about mongre2 on the official website: http://mongrel2.org.

The key feature of mongrel2 is that, that fact of using a queue to communicate with the applications makes it possible to run the apps decoupled from the server, this means that your app doesn’t run with a mod_something or as a child process of the webserver. Another advantage is that ØMQ allows you to communicate over TCP, that’s how you can run your app on a machine other than where mongrel2 is running.

The ability to run your app this way, that is, on different machines is fantastic, because whenever needed you can add a new node to your cluster and start new instances of your app. All these instances will connect to the same mongrel2, and the requests will be load-balanced among all instances in a round-robin policy.

wsgid

wsgid is a project I developed to make possible to run your WSGI apps with mongrel2 webserver. wsgid is the brifge between mongrel2 and the WSGI specification (PEP-333), this means that just by conforming with the WSGI spec your app is ready to run with mongrel2+wsgid.

wsgid talks both mongrel2 and WSGI, also from now on your app will be a separated process, with it’s own PID and everything a process has. This brings you some advantages:

  • Can run as any user on your operating system;
  • As being a O.S process, is automatically submitted to all characteristcs and O.S process has, eg: bandwith limit, memory limits, CPU scheduling, etc;
  • Can run inside a chroot, in caso you are running untrusted code;
  • Many others.

Start new instances of your app is as easy as:

   $ wsgid --app-path=/path/to/wsgid-app-folder/ --recv=tcp://127.0.0.1:8888 --send=tcp://127.0.0.1:8889
      --workers=4

I this very simple example we are connecting to a ØMQ endponit in same machine that the app will be running, but nothing prevents us from using –recv=tcp://192.168.0.2:8888, being this IP the address of another node on the cluster. Still in this example, the option –worker=4 starts 4 processes that will respond to requests for your app.

wsgid, key features

Besides facilitating the deployment of WSGI applications with mongrel2, wsgid also has other important features:

workers

With this option you will be able to start any number of processes you want at once. That means you don’t need to run the same wsgid command 4 times if you want four instances, although you can do this without any problem. But this option gives you the possibility to do –workers=4, which gives you the same result.

keep alive

With the keep-alive option wsgid automatically restarts any process which has finished its execution. This means that with a call to wsgid with –workers=4 –keep-alive, wsgid will always keep 4 processes working for your application.

hot deploy

Whenever wsgid re-starts one of the processes, the code of your application is reloaded, after all is a new process being created. That means you can update the source code of your application and just SIGTERM all its workers, it will make  wsgid restart all workers, but this time already running the new source code.

chroot

wsgid can chroot to the location where the application is. This makes such application run in isolated, thus creating an environment a little safer (not entirely) if you are running untrusted code.

Plugable system for Application Loading

wsgid has a very simple but very powerful plugin sub-system . With it you can write your own application loader, in case wsgid is not able to load your WSGI app. With such a sub-system, it will be very simple to add support for other WSGI frameworks so that more applications can make use of this project.

That’s it! Hope you enjoyed the project, any feedback is more than welcome. To learn more about the project visit the official website: http://wsgid.com.

But remember, use what works best for you, use what best serves you. I’m not saying that the solution mongrel2+wsgid is the best ever, I’m just saying that this combination is fantastic and should be considered whenever you are thinking of publishing a new WSGI application.

Just out of curiosity, the wsgid.com website is a django application and runs with wsgid.

,

2 Comentários

Plug n’ Play: um sistema de plugins genérico para python

Nesse post veremos um projeto open source chamado Plug n’ Play. É um sistema de plugins genérico para python. Isso significa que com esse projeto é possível desenvolver de forma modular e também adaptar códigos existentes para que, a partir de então, seja possível adicionar novas funcionalidades sem modificar o código principal do programa.

Leia o resto deste post »

2 Comentários