Frequently Asked Questions about SmartFrog
What is SmartFrog?
SmartFrog is a software framework for helping to build distributed, component-based software systems in a way that makes them easy to configure, automatically install and start, and automatically shut down. Such systems typically have multiple software components running across a network of computing resources, where the components must work together to deliver the functionality of the system as a whole. It's critical that the right components are running on the correct computers, that the components are correctly configured, and that they are correctly combined together into the complete system. This profile fits many of the services and applications that run on today's computing infrastructure.
Concrete examples are three-tier web applications, which often consist of a database server, application logic middleware, web server software, firewalls and load-balancers. All of these can be thought of as components that need to work together to deliver the complete web-service. Each component must be installed on an appropriate resource and correctly configured. Components must be started in a certain sequence, and linked together into the complete system.
With SmartFrog, system configuration details such as: which components make up the system, on which computers those components should run, how the individual components are configured, in what order they are started, and how they are combined into a complete system, are captured in "system descriptions". These descriptions, written in the SmartFrog description language, are interpreted by the SmartFrog distributed runtime engine in order to install the required software components, to configure them, and to start the complete software system according to the details of the description.
What advantages does SmartFrog offer?
First, SmartFrog allows you to take all the details of a software system that can be considered configuration-related and record them in one place as part of a SmartFrog system description. This means it is very easy to make changes to the configuration of the system and hence change the operation of the system when it is installed and started. The description method also allows systems to be built from pre-defined templates, and allows systems to be constructed from sub-systems. We call such systems "configuration-driven".
Secondly, given the SmartFrog descriptions, systems can be installed, started, stopped and uninstalled completely automatically. This means that a given system description can be installed automatically, with multiple instances, and that this can be done repeatably and correctly.
This functionality makes SmartFrog ideal for utility computing applications - which need to be automatically installed and started on utility computing resources, repeatedly, and removed cleanly when they are no longer needed.
What technologies does SmartFrog consist of?
There are three main parts to the SmartFrog framework. The first is a configuration description language to define the configuration data appropriate to the SmartFrog components. The language has many convenient features for describing configuration data, and provides the notions of abstraction and template creation which are necessary to build complex configuration definitions.
The second part is a "component model"; a specification of how SmartFrog components are written and integrated into the SmartFrog framework. This includes aspects such as the lifecycle of a SmartFrog component and mechanisms for checking what state the component is in (such as whether it is still "alive"). The components are responsible for interpreting the configuration details given in the SmartFrog Language and then carrying out the necessary configuration tasks base on these details. SmartFrog includes some standard components - offering features such as component sequencing, distributed workflows, failure detection, script execution, installation of software, dynamic discovery and naming -and new ones can easily be added for new applications.
The third part is the runtime system, made up of SmartFrog "daemons" that run on every element in the network. The runtime system interprets SmartFrog system descriptions and uses them to control the creation of components in the right location, with the correct configuration parameters. It also provides facilities for tools to locate and interact with components, for monitoring their lifecycles, and for carrying out other runtime tasks. It is fully distributed, with no required central point of control and therefore scales very well. Additionally, because of the fully-distributed nature of the runtime system, any component can use it directly to create and manage its own sub-systems as part of a dynamic and adaptive system configuration infrastructure.
In addition to the specific parts of the framework, SmartFrog has a methodology a way of thinking about the problem for extending the set of key elements in a consistent style to cover areas not yet handled by the framework.
You say that SmartFrog is a "framework"; what do you mean by this?
This is intended to indicate two things: first that SmartFrog is not a packaged product or solution; rather it is a building block to help build configurable software systems. So, while it has wide applicability, new SmartFrog components will usually need to be developed for each new application. Secondly, that SmartFrog can be extended by "plugging in" new components and system descriptions into the framework. (In fact, the system itself is built out of components that satisfy its own component model, so that many system behaviours can be changed by replacing or modifying the existing components.)
Where was SmartFrog developed?
SmartFrog was created in HP Laboratories as part of a research project working on configuration management for domains ranging from distributed measurement and management systems, through to dynamic configuration of adaptive services on utility computing infrastructures such as those offered by the Grid.
How secure is SmartFrog?
Without security, SmartFrog would allow anyone to deploy programs on remote machines! Fortunately, we take security seriously, and the system protects itself from malicious use using a public key infrastructure (PKI) system. Each node that participates in a SmartFrog runtime system is supplied with a certificate. Furthermore, all software components and system descriptions are "signed" with a certificate. The certificates are used to permit only validated nodes to participate in the runtime system, and those nodes will only manipulate components and descriptions that have been appropriately signed. Additionally, all network communication takes place using an encrypted transport. Currently there is a single-level security model, where nodes and components are either fully trusted or not trusted at all. Finer grained models of security are currently being researched.
What does the name "SmartFrog" stand for?
SmartFrog stands for "Smart Framework for Object Groups", which captures the notion that this is a framework rather than a complete product, and that its purpose is to intelligently orchestrate the activities of groups of software objects or components.
How is SmartFrog implemented, and what platforms does it run on?
The core SmartFrog system is implemented completely in Java, and therefore runs on any platform that supports a valid Java environment. We have used SmartFrog on Microsoft Windows, Linux and HP-UX. Some SmartFrog components are designed to work on specific operating systems, or to interface with specific technologies that have platform-specific requirements, so these are more limited in their portability.
Do I have to completely rewrite my applications as SmartFrog components in order to use SmartFrog?
No. In fact it's generally fairly easy to write simple SmartFrog components that "wrap" or encapsulate non-SmartFrog things, be they in Java or native code. For example, we've done this with the Apache web server where we created simple SmartFrog components that are capable of installing, configuring, starting, stopping and uninstalling the Apache system. This allows us to completely manage the Apache system from SmartFrog - to install it on a specific system, to configure it correctly, to start and stop the system and to connect it in to other system components, all of which can be driven from a system description in the SmartFrog language.
One caveat about this approach is that it does rely on the encapsulated system being cooperative - in the sense that controlling it must be possible via an API or command line scripts. However, if everything to be done through a graphical user interface (GUI), this can cause problems for the types of automation that are required.
What is the status of SmartFrog?
SmartFrog has been developed over a number of years, and has been used as an embedded component in a product context. It is in active use as a framework for experimentation. Our recent work has focused on making the system general-purpose and on extending our experience of using it to configure and deploy a range of services and applications. We are now at the stage where want to work with external partners to co-develop the technology and the approach, so we have released the system under an open source licence.
So it's open source; what's the licence?
We are releasing the system under the LGPL open source licence. This means that third parties can use the SmartFrog framework freely for commercial or non-commercial purposes, and can build proprietary, private components and templates that use the framework. However, they must acknowledge the use of the system, must provide access to any core SmartFrog source code, and must publish any changes to the SmartFrog core libraries under the same LGPL licence.
Where and when can I get the system?
The system is available right now from SourceForge
What are the costs and benefits of using SmartFrog?
There is a cost to using SmartFrog. The largest element of cost is that of initially learning the intricacies of the framework and its methodology since the model of configuration is probably unfamiliar. Once familiarised, though, the cost of creating new SmartFrog components is in direct proportion to the complexity of the task with little or no overhead imposed by the use of the framework. Indeed, the task is often actually simplified, as one of the key aspects of the component model is that the programmer's interface to configuration data is consistent regardless of how the data is to be obtained - from a static text file through to being dynamically discovered at runtime. The SmartFrog framework hides the precise mechanism being used, and allows new mechanisms to be easily incorporated.
The largest benefit comes once components have been designed and implemented. The flexibility and ease of using SmartFrog components is greatly enhanced over the more traditional model of using configuration scripts and installers, or various ad-hoc models of lifecycle management that might be developed for a specific purpose. These benefits come from the strong separation between configuration data and the functional parts of the components, plus the provision of a rich variety of ways of collating the configuration data.
Consequently, many different configuration models may be implemented, from static data defined in configuration descriptions through to dynamic models using naming or discovery services. These may all be implemented without the core configuration components having any knowledge of how the data is defined embedded in the component code. In practice, this has been shown to greatly improve component reuse and allows component developers to greatly simplify their tasks.
Name the most important benefits of using SmartFrog
Key benefits of automating deployment with SmartFrog are:
Is there anything else like SmartFrog available today?
There are many software configuration systems, but none that take exactly the same approach. Most products fall into one of two classes of system: image installers and software distributors. The image installers use boot-time protocols to install an operating system image. The software distributors, are mostly targeted at distributing software and OS patches to running computers according to some overall policy defined in some database. Neither of these two approaches handle the runtime aspects of lifecycle management, those relating to dynamic binding, rapid reconfiguration to cope with loading or failure, and so on.
The closest product to SmartFrog today is Puppet, the Ruby-based tool from Luke Kanias from Reductive Labs. They are both declarative systems, with languages designed to enable sharing of configuration information across deployed components.
What is a component?
A component is something that SmartFrog deploys. In a deployment, SmartFrog deploys the parent component of the application, which then can deploy its children in the manner of its own choosing. The basic
What do components do?
Whatever they want! Generally, a component does the following things
A key feature here is that each component does more than just configure and start its code, it checks up on its own health, and knows how to terminate its running processes or threads when requested. This lets SmartFrog assess the entire health of an application, and, when terminating an application, means that it can stop the entire application.
What is an application?
An application is an aggregation of components. For a Java application server web server with a database back-end, that could include:
A more complex configuration may take this and extend it to include
These settings can be added on top of the existing configuration through aggregation -by taking one application and including it in another. Configuration details such as usernames and ports could be shared, so that the application server would listen on the same port the firewall opened. The web page health checker component would use the same port information, and the network address of the application server -the latter being a LAZY attribute added by the application server at run time
What is liveness?
In distributed computing terminology, liveness is a measure of the health of part of the system. In SmartFrog, a component is live if it responds to a liveness test (an
A Compound responds to a liveness request from its parent by checking the health of all its children: it is live only if its children are live.
Individual components are free to construct their own liveness tests. The web page checker component acts by polling the web page it has been bound to, and verifies that the response code is within a permitted range, and that the size and MIME type the response are within specified ranges. Such components blur the distinction between a deployment framework and a management framework -you can not only declare what components constitute an application, you can declare some of the checks for the health of the deployed application.
There are a number of advantages of including such information into the application:
The user must use the SmartFrog notation to define a number of aspects of the configuration - in particular which SmartFrog components to use, their configuration data (or how to obtain it), the relationships between the components (such as start-up order, etc.), where they should be installed and started, and so on. Specific descriptions, parameterised by a number of configuration attributes, or patterns for such descriptions, may be captured in templates that may be used and extended for more specific purposes.
There are currently no GUI tools supporting the development of templates or final descriptions. Such tools are possible and this is an area of active research. However the complexity of large systems is such that GUI-based tools are unlikely to scale well - they work best in limited domains and so supporting GUI tools are likely to be frameworks to allow users to easily develop a specialised tool to go with specific templates. This makes it easier to release templates to end-users who can see high-level abstracted models rather than have to work with the details. A general purpose GUI editor is probably of little use.
How robust is SmartFrog?
The core framework has been developed over several years, and has been used for developing many services. Consequently, it is considered to be fairly robust by regular users. We have a lot of functional tests for all the different components, tests we run by deploying components and tests on different machines, using SmartFrog itself. A distributed deployment framework is the ideal infrastructure for testing distributed deployments!
Where can I get more information?
Start at http://www.smartfrog.org/, and use the contact methods listed there if you need to get in touch.
What are the platforms supported by SmartFrog?
SmartFrog is written in Java, and runs on supported on following Operating Systems:
Which JDK version is required for SmartFrog?
Java 5 or later or above is required to run SmartFrog.
Are there examples for using SmartFrog?
SmartFrog provides a set of examples with the release. These examples are intentionally kept small and simple. SmartFrog also provides web interface for running the examples. Please refer to
What are a
A component that can have children
A component that can listen to and raise events as part of a workflo
A component that can have children as part of a workflow. This compound allows subclasses to manage the lifecycle of its children in ways more advanced than the basic Compound allows.
Why I should implement Prim?
Every component that is remotely accessible needs to directly declare that it implements a
java.rmi.Remote interface, or one that derives from it, which
Prim does. Without this, the
rmic compiler will not generate remote stubs, and the class/component will not be accessible remotely.
What is needed in a remote machine to deploy SmartFrog components?
- A Java VM
- SmartFrog itself
- the correct set of security certificates needed to authenticate communication and JAR file loading
- Either the JAR files for every component, or access to a server that is providing them through dynamic class downloading
- A Firewall open at the port that SmartFrog is listening for RMI connections. By default, this is port 3800
- SmartFrog running under the account that you wish to use to deploy components.
How can I get an attribute of a component?
Attributes of a component are stored in a
Context. To access this attributes use the
sfResolve method of the Prim interface. This will not only resolve an attribute, it will follow any references to return the final value - or throw a SmartFrogResolutionException if the resolution failed.
How can I modify an attribute value?
There are three methods to do that:
sfAddAttribute( Object name, Object value): adds an attribute to the component's context. If an attribute already exists with this name it is not replaced;
sfRemoveAttribute( Object name ): removes the named attribute from the context;
sfReplaceAttribute( Object name, Object value ) adds or replaces the named attribute. This is the normal method used to set an attribute.
How can I get an attribute of a component?
You can use
sfResolve(), once you have the Prim interface to that component. There is a static method
SFProcess.getProcessCompound() to give you the root compound of the current process:
If you need to get an attribute that is in another machine:
First, get the root process of the remote machine:
SFResolve(Reference.fromString("HOST hostname")), then request the attribute
Is it possible chose a different deployer for my application?
Yes. SF provides three different deployers: PrimDeployerImpl, PrimHostDeployerImpl and PrimProcessDeployerImpl. The last one is the default deployer. The first one is used to deploy the component in the current JVM, ignoring any other attributes. The second one deploys the component in the root process compound of the host whose name is provided by the sfProcessHost attribute. The last one deploys the component in the named process of a specified host. To chose the deployer it's necessary set the sfDeployerClass attribute of the component. To each component is possible chose a different deployer.
How do I change the default Log name for my component?
You just need to add the attribute sfLog to your component.
How do use traditional class name for my component logging?
Just add the following line to your component description:
How can I send my logs to SysLog?
You could use the Log4J adaptor.
default.ini to use the Log4J adaptor:
Add the follwing configuration to your
Example filters for syslog_ng:
Info about syslog_ng: What is with all the "hostname" options ( http://www.campin.net/syslog-ng/faq.html )?
When syslog-ng receives a message it tries to rewrite the hostname it contains unless keep_hostname is true. If the hostname is to be rewritten (e.g. keep_hostname is false), it checks whether chain_hostnames (or long_hostname which is an alias for chain_hostnames) is true. If chain_hostnames is true, the name of the host syslog-ng received the message from is appended to the hostname, otherwise it's replaced.
So if you have a message which has hostname "server", and which resolves to "server2", the following happens:
Unable to connect to the SmartFrog Daemon
I've tried to deploy my application using sfStart, but I get the following message : Unable to connect to sfDaemon on: localhost. Reason:sfDaemon may not be running on localhost java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested excepti on is: java.net.ConnectException: Connection refused: connect What's wrong ?
The daemon is not started on the localhost. Launch the daemon on the localhost and then re-deploy the application.
Unable to deploy my application using
sfStart: I've tried to deploy my application using
sfStart localhost ex1 myapp.sf, but I get the following message : Unable to locate File or URL: myapp.sf for component : ex1 Reason: The path to URL may be incorrect or file may be missing.
What should I do ?
myapp.sf does not exist. Make sure that the filename is correct and the path of the file is correct or the jar file containing the file myapp.sf is in the classpath.
Duplicate component instance: I've tried to deploy my application using
sfStart localhost myApp example\myFile.sf but I get the following message :
What's wrong ?
Some other application with the name
myApp is already deployed on the root daemon of the localhost. Deploy the application
example\myFile.sf with a different application name. If you wish to update an existing application by that name, use sfUpdate, or stop the application with sfTerminate and redeploy the new version.
Not able to start SmartFrog daemon with security on
I've set up the security infrastructure, signed the jar files, and build the certificates. The daemon does not start. What have I done wrong ?
Transfer the signed jar files and the host keys and properties files securely to the host and then start the daemon. One source of error might be an incorrect classpath, especially the paths for some of the required files (sf.policy or mykeys.st or SFSecurity.properties) Set these two paths correcly.
No deployment when security is turned on
I've set up the security infrastructure, and started a secure daemon. However when I try to deploy a description, I get a class loader exception. What's wrong ?
The classloader, when looking for the file, will look in the classpath. Since the classpath only contains the signed jars make sure that your application's SmartFrog description is in the signed jar.
Not able to remotely deploy application with security on
I've managed to start two daemons with security on, on two machines. However when I try to deploy an application across the two machines it does not work. What should I do ?
The classloader when looking in the classpath may find different signed jars.Make ure that the two machines have the same signed jar files.
How do I keep track of the deployed components in a host?
Use the sfManagementConsole UI tool to keep track of the components deployed in a host.
I get ClassNotFoundException when I try to deploy a component on a remote host from localhost. However my localhost has all the required JAR files
When a component is deployed on a remote host the process compound on that host drives the deployment of children components, and thus looks for the required classes on the remote host. If the remote host does not have the required jar files it results in ClassNotFoundException.
Action: Put the required jar files on the remote host and deploy the component again.
How to get a JVM thread dump:
The Thread Dump
... is a textual dump of all active threads and monitors of Java apps running in a Virtual Machine.
To generate thread dump:
Press Ctrl-Break in the command console you used to start the jvm
- Unix, Linux, Mac OS X
Press Ctrl-\ in the terminal console you used to start the jvm
Send the QUIT signal to the Java VM running the IDE
kill -QUIT process_id
process_id is the process number
In Unix you can also invoke kill -3 on the Java process, which dumps the thread dump to the console that is running the Java program.
- jstack tool
... is a monitoring tool that is part of JDK installation. The documentation:
jstack from JDK1.5.0 -> http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html
jstack from JDK6.0 -> http://java.sun.com/javase/6/docs/technotes/tools/share/jstack.html
On Mac OS X, jstack sometimes fails with an error message. As a workaround, you can generate a thread dump using killall -9 hsdbd; sudo jstack pid
- StackTrace tool
... allows to get thread dump even without console. It works on all platforms and for all JDKs.
DOS/Windows Shell Scripts
How do I redirect "Standard Output and Error" to a file?
Use: command > file 2>&1Ex. smartfrog > log.txt 2>&1
command > file
Write standard output of command to file
command 1> file
Write standard output of command to file (same as previous)
command 2> file
Write standard error of command to file (OS/2 and NT)
command > file 2>&1
Write both standard output and standard error of command to file (OS/2 and NT)
command >> file
Append standard output of command to file
command 1>> file
Append standard output of command to file (same as previous)
command 2>> file
Append standard error of command to file (OS/2 and NT)
command >> file 2>&1
Append both standard output and standard error of command to file (OS/2 and NT)
commandA | commandB
Redirect standard output of commandA to standard input of commandB
commandA 2>&1 | commandB
Redirect standard output and standard error of commandA to standard input of commandB (OS/2 and NT)
command < file
command gets standard input from file
command's standard error is redirected to standard output (OS/2 and NT)
command's standard output is redirected to standard error (OS/2 and NT)
More info: http://www.robvanderwoude.com/redirection.html|