If you use OSGi in order to make your application dynamic, then your tests should test the dynamicity aspect of your application, otherwise how would you know whether your application behaves dynamically or not? how would you be sure that clients use the service with highest rank when such is registered? that bundle updates wouldn't break the application? or that any other risks you have when working on dynamic OSGi-based applications are tested and verified properly? Surely, you wouldn't want to verify this behavior in production systems or manually. Agreeing on the importance of dynamicity tests, we will move to the next issue that developers of dynamic OSGi-based applications have, how to test dynamicity?

Current Approach for Testing Dynamic Applications (xUnit based)

Currently, all of the approaches, be they provided by OSGi vendors, like test support classes that ship with Spring-DM or any other custom efforts, revolve around the idea of running an OSGi Application inside JUnit tests then asserting different parts of the OSGi-based application using the Bundle Context. Below is an example test which verifies that a service which implements the AccountsService interface is registered after installing a bundle.

@Test
public void accountServiceVerifactionTest() throws Exception {
	/// We assume that you've started the OSGi Environment either by
	/// running it on your own or using helper or super classes.
	BundleContext bundleContext = getBundleContext();
	
	/// Here you use some code you developed to help you locate
	/// the bundles you are testing
	URL accountingBundleUrl = locateBundleBySymbolicName(
			"org.test.accounting", "1.0.0");
	
	bundleContext.installBundle(accountingBundleUrl.toString()).start();
	
	ServiceReference accountServiceRef = bundleContext.getServiceReference(
			"org.test.accounting.AccountService");
	
	assertNotNull(accountServiceRef, "AccountService was not registered");
	
	/// You can't execute the code below since it will throw a Class Casting
	/// exception. This is due the fact that the OSGi Enviornment uses a 
	/// different Class Loader from the one JUnit uses
	AccountService accountService = bundleContext.getService(accountServiceRef);
}
	

The approach above has few critical issues that can turn OSGi-based application testing experience into nightmare:

• Class casting problems: You can't invoke objects and services provided by the OSGi Environment in a simple way because the OSGi Environment and JUnit use different class loaders. Accessing objects of the OSGi Environment from a non-OSGi application is not a critical requirement at most of the cases, but it surely is when testing OSGi applications, otherwise how would you verify that your services perform correctly.
• Readability: This might not seem an issue in the simple example above, but when we will tackle a more complex case below, you can spend some time imagining how it will look using the approach above. Since in real world applications we will have at average tens of OSGi Services per application, the readability aspect is critical to help us maintain high quality tests.
• A lot of maintaining code: Much of the code is responsible for OSGi Environment initialization, bundle archives location logic, etc. and has nothing to do with measuring changes. Using some libraries might reduce the maintaining code, but it's still a time consuming issue that needs to be dealt with.

DA-Testing with a Dynamic Oriented and OSGi-friendly approach

DA-Testing, the first testing framework intended for testing OSGi-based applications, moves away from the xUnit approach to a dynamic oriented one for testing dynamic OSGi-based applications. Why? Well, because we don't test units, be they classes or methods, but the dynamic behavior of the OSGi-based application. We apply runtime changes to the application and check how different components react to them. We don't ask questions like: What the result I will have if I pass these parameters? We ask questions like: If I remove this bundle or service, will the dependent components still perform normally? If I install a newer version of this bundle, will the behavior of the application change? or will it stay the same? Dynamicity is all about changes and reacting to them, as such, the testing approach we use should provide us with means that simplify the task of testing these changes.

DA-Testing helps developers testing dynamic OSGi-based applications by implementing the concepts below:

• Dynamic Oriented Tests Structure: Tests with DA-Testing are structured and executed in a dynamic oriented fashion. Tests are not executed randomly or sequentially, but as reactions to changes in the OSGi Environment.
• Tests Run in the OSGi Environment: Test-related components are grouped into a Test Bundle and run in the OSGi Environment. This way, tests benefit from the capabilities provided by the OSGi Environment and no class casting problems need to to be handled by developers.
• OSGi-friendly API for testers: DA-Testing provides an API for asserting OSGi-related components, easily locating and installing bundles, etc. and uses the capabilities provided by Java 5 to simplify working on core OSGi tasks.

Instead of explaining the concepts as a set of abstractions, we will tackle a task of testing a not very simple example application with DA-Testing.

Testing Dynamic Store Application with DA-Testing

The Dynamic Store is a simple application that provides four services:

• Data Storage service: Persists entities used by the application (in a database for example).
• Orders service: Allows us to Place New Orders and Find Client Orders. It uses the Data Storage service to save and retrieve orders. If the Data Storage service is unavailable when an order is placed, the order will be queued until the Data Storage is available, then it will be moved there. Unlike placing new orders, finding client orders method will not be available if the Data Storage service is not, it will throw a ServiceUnavailableException.
• Welcoming service: Provides a Client Welcoming message. The developer of this service made a typo, as a result the service returns the message "Stay away, lousy clients!" instead of "Welcome, our lovely clients!". We will provide a patch to fix this bug.
• Web Resources service: Uses the Welcoming service to provide the contents of an HTML page.

The application is designed to be flexible so it could be updated and patched easily, as such, we have the following bundles:

• Data Store API: Provides services interfaces and entities.
• Data Store Core: Implements service interfaces and provides them as OSGi services. It doesn't provide a DataStorage service since in production systems we usually provide services like the Data Storage one in separate bundles for flexibility issues.
• Data Store Core Patch: Provides a new implementation of the Welcoming Service as an OSGi service with a service rank higher than the Welcoming service provided by the Core bundle.

Each of the bundles above plus the OSGi Tests bundle has its Maven project. Please, note that the current release of DA-Testing is runnable only with Maven. Also, I removed all the comments and description text from the demonstrated code, if you want to read them, please refer to the source code.

So, what we want from our tests?

• To check how the Orders service reacts to Data Storage service availability changes.
• To confirm that if we install the Patch Bundle, the Web Resources service will use the new Welcoming service instead of the one provided by Core bundle. And if the patch is rolled back (the bundle is uninstalled), the Web Resources service will move back to the old one.

Testing Data Storage service availability changes

Defining requirements for dynamicity of our application, we always think of Dynamicity Scenarios. Defining dynamicity requirements we always say something like: What if service A became unavailable for a moment, how should dependent service B behave? If we provide a patch of bundle A which if its components need to be updated? etc. We define our dynamicity requirements this way, and so should be our tests, that's why the first thing we do when testing applications with DA-Testing is creating Test Scenarios (Dynamicity Scenarios). Now, lets see how the Test Scenario that emulates Data Storage service availability changes would look like:

@OsgiTestScenario(name = "Data Storage Availability Test Scenario",
		description = "...",
		testGroups = {
				NormalServicesBehaviorTests.class, 
				DataStorageAvailabilityTests.class })
public class DataStorageAvailabilityTestScenario extends AbstractOsgiTestScenario {
	
	@Override
	public void prepare() throws Exception {
		installAndStartBundles(getBundleStore().findBySymbolicNames(
				SymbolicNames.DYNAMIC_STORE_API,
				SymbolicNames.DYNAMIC_STORE_CORE));
	}
	
	@Override
	public void perform() throws Exception {
		registerService(DataStorage.class, new DataStorageServiceMock());
	}
	
	@Override
	public void finish() throws Exception {
	}
	
}

A Test Scenario in DA-Testing has three parts: prepare, perform and finish. Before performing a scenario, we prepare the OSGi Environment, and that's exactly what the prepare method does, we installed the API and Core bundles and started them, as you can see we installed these bundles in a single line of code (broken into 3 lines for the sake of clarity) without having to locate the bundle archives, also, DA-Testing will generate bundles from the OSGi-incompliant bundles if such exist (you can imagine how much time only these two features will save you). Any code executed in the prepare and finish methods is transparent to our tests, as such they will never be invoked based on changes done in these methods. So far so good, we have an OSGi Environment with two bundles, the API and the Core, but without a Data Storage service, while performing the scenario we register a mock Data Storage service. As you see the code is quite clean, the scenario code, the one which emulates the changes we want to see how our application reacts to, is completely separated from test assertions, as such, we can clearly express our dynamicity scenario in code.

Now, we want to check how the application behaves when the Data Storage service is available and when it's not. You probably noticed that the @OsgiTestScenario annotation has the testGroups attribute, this attribute is responsible for specifying the classes that contain the tests which are responsible for testing application reaction to changes resulted from performing the Test Scenario. The DataStorageAvailabilityTests class has tests that check how the Order service reacts to Data Storage availability changes. Prior to demonstrating the full code of the DataStorageAvailabilityTests class, lets examine a single test which verifies that the placeOrder method accepts requests even when the Data Storage is unavailable:

	@OsgiTest(name = "Place Order Async Processing Test",
			description = "Place Order method of the OrderService must accept" +
					" requests even when the DataStorage service is not available.")
	@Conditions(serviceStateConditions = {
			@ServiceStateCondition(serviceClass = ClassNames.ORDER_SERVICE,
					state = ServiceState.AVAILABLE),
			@ServiceStateCondition(serviceClass = ClassNames.DATA_STORAGE,
					state = ServiceState.UNAVAILABLE)
	})
	public void testPlaceOrderAsyncProcessing() {
		getService(OrdersService.class).placeOrder(new Order(999, 999));
	}

As I mentioned before, tests are not executed randomly or sequentially, but in response to changes and the state of the OSGi Environment, as such, each test in DA-Testing has a condition annotation which specifies under which circumstances the test needs to be executed (you can also provide tests with no condition annotation, then it will be executed as soon as the Test Scenario performance begins). For instance, the test above is executed when the Order service is available and the Data Storage service is not, this happens in our Test Scenario prior to registering the mock Data Storage service. As you can see we acquired a reference to the Orders service and invoked the placeOrder method in a single line of code, another example of how the API is meant to simplify your life. Now, lets take a look at the full code of DataStorageAvailabilityTests class:

@OsgiTestGroup(name = "DataStorage Availability Tests")
public class DataStorageAvailabilityTests extends AbstractOsgiTestGroup {
	
	@OsgiTest(name = "Place Order Async Processing Test", description = "...")
	@Conditions(serviceStateConditions = {
			@ServiceStateCondition(serviceClass = ClassNames.ORDER_SERVICE,
					state = ServiceState.AVAILABLE),
			@ServiceStateCondition(serviceClass = ClassNames.DATA_STORAGE,
					state = ServiceState.UNAVAILABLE)
	})
	public void testPlaceOrderAsyncProcessing() {
		getService(OrdersService.class).placeOrder(new Order(999, 999));
	}
	
	@OsgiTest(name = "Find Client Orders method unavailability Test",
			description = "...")
	@Conditions(serviceStateConditions = {
			@ServiceStateCondition(serviceClass = ClassNames.ORDER_SERVICE,
					state = ServiceState.AVAILABLE),
			@ServiceStateCondition(serviceClass = ClassNames.DATA_STORAGE,
					state = ServiceState.UNAVAILABLE)
	})
	public void testGetOrdersUnavailability() {
		/// Although DA-Testing provides @ExpectedException annotation, currently,
		/// it's usable only with general exceptions (not application specific). 
		/// This is due the fact that application specific exception classes are 
		/// available only after executing test scenarios, which causes class 
		/// loading conflicts when generating tests. Using the exception in 
		/// catch block will cause the same problem.
		try {
			getService(OrdersService.class).findClientOrders(1);
			
			throw new ExpectedExceptionNotThrownException(
					ServiceUnavailableException.class);
		} catch (RuntimeException ex) {
			if (!(ex instanceof ServiceUnavailableException)) {
				throw new ExpectedExceptionNotThrownException(
						ServiceUnavailableException.class);
			}
		}
	}
	
	@OsgiTest(name = "Orders data synchronization Test", description = "...")
	@Conditions(serviceStateConditions = {
			@ServiceStateCondition(serviceClass = ClassNames.ORDER_SERVICE,
					state = ServiceState.AVAILABLE)
		}, serviceEventConditions = {
			@ServiceEventCondition(serviceClass = ClassNames.DATA_STORAGE,
					event = ServiceEventType.REGISTERED)
	})
	public void testOrderDataSynchronization() {
		/// We will give the OrderService some time to fill the 
		/// DataStorage service that was registered
		sleep(50, MILLISECONDS);
		
		/// We check that the order we created before DataStorage was 
		/// available is moved to the storage
		getAssertions().assertTrue(
				containsOrder(getService(DataStorage.class), 999, 999));
	}
	
	
	protected boolean containsOrder(
			DataStorage dataStorage, int clientId, int productId) {
		for (Order order : dataStorage.getOrders()) {
			if (order.getClientId() == clientId
					&& order.getProductId() == productId) {
				return true;
			}
		}
		return false;
	}
	
}

DataStorageAvailabilityTestScenario uses two Test Groups, I didn't cover the second since it performs simple tests that verify the behavior of the application in normal circumstances and repeats much of what was presented above.

Testing the Patch Bundle

The test of the Patch Bundle should confirm that the application changes its behavior when we install the Patch Bundle and is capable of behaving as it used to do if we decide to roll back the patch. For that we created one Test Scenario PatchInstallationTestScenario which will install the patch then uninstall it, the assertions are in the Tests Group PatchInstallationTests. Below is the example code:

@OsgiTestScenario(name = "Patch Installation Test Scenario",
		description = "...",
		testGroups = {
				NormalServicesBehaviorTests.class,
				PatchInstallationTests.class })
public class PatchInstallationTestScenario  extends AbstractOsgiTestScenario {
	
	@Override
	public void prepare() throws Exception {
		installAndStartBundles(getBundleStore().findBySymbolicNames(
				SymbolicNames.DYNAMIC_STORE_API,
				SymbolicNames.DYNAMIC_STORE_CORE));
		
		registerService(DataStorage.class, new DataStorageServiceMock());
	}
	
	@Override
	public void perform() throws Exception {
		/// We install the patch
		Bundle patchBundle = installAndStartBundle(
				getBundleStore().findBySymbolicName(
						SymbolicNames.DYNAMIC_STORE_CORE_PATCH));
		
		// Then remove it
		patchBundle.uninstall();
	}
	
	@Override
	public void finish() throws Exception {
	}
	
}

@OsgiTestGroup(name = "Patch Installation Tests")
public class PatchInstallationTests extends AbstractOsgiTestGroup {
	
	@OsgiTest(name = "Home Page HTML Verification Test", description = "...")
	@ServiceStateCondition(serviceClass = ClassNames.WEB_RESOURCES_SERVICE,
			state = ServiceState.AVAILABLE)
	public void testHomePageHtml() {
		getAssertions().assertEquals("Stay away, lousy clients!",
				getService(WebResourcesService.class).getHomePageHtml());
	}
	
	@OsgiTest(name = "New Welcoming Service Test", description = "...")
	@Conditions(
			/// Of course we can skip on the Bundle State condition, but 
			/// I put it here for demo purposes
			bundleStateConditions = {
					@BundleStateCondition(
							symbolicName = SymbolicNames.DYNAMIC_STORE_CORE_PATCH,
							state = BundleState.STARTING)
			},
			serviceEventConditions = {
					@ServiceEventCondition(
							serviceClass = ClassNames.WELCOMING_SERVICE,
							serviceFilter = "(updateReason=patch)",
							event = ServiceEventType.REGISTERED)
	})
	public void testNewWelcomingService() {
		getAssertions().assertEquals("Welcome, our lovely clients!",
				getService(WelcomingService.class).welcomeClients());
	}
	
	@OsgiTest(name = "Home Page after New Welcoming Service Availability Test",
			description = "...")
	@ServiceStateCondition(serviceClass = ClassNames.WELCOMING_SERVICE,
			serviceFilter = "(updateReason=patch)",
			state = ServiceState.AVAILABLE)
	public void testHomePageHtmlAfterNewWelcomingServiceAvailability() {
		/// We give the WebResourcesService a moment to use the new 
		/// version of the Welcoming Service
		sleep(50, MILLISECONDS);
		
		getAssertions().assertEquals("Welcome, our lovely clients!",
				getService(WebResourcesService.class).getHomePageHtml());
	}
	
	@OsgiTest(name = "Home Page after New Welcoming Service removal Test",
			description = "...")
	@ServiceEventCondition(serviceClass = ClassNames.WELCOMING_SERVICE,
			serviceFilter = "(updateReason=patch)",
			event = ServiceEventType.UNREGISTERING)
	public void testHomePageHtmlAfterNewWelcomingServiceRemoval() {
		/// We give the WebResourcesService a moment to move back to 
		/// the old version of the Welcoming Service
		sleep(50, MILLISECONDS);
		
		getAssertions().assertEquals("Stay away, lousy clients!",
				getService(WebResourcesService.class).getHomePageHtml());
	}
	
}

As you can see, other than the clear separation between dynamicity scenario and test assertions, the Bundle location logic, accessing OSGi objects, services invocation, checking conditions of the OSGi Environment, bundle generation, listening to events, etc. is all transparent/simplified for you. If you are not sure of the benefits you would be getting from using DA-Testing, you can test the same application with the current tool/approach you are using and compare code size and development time to achieve the same result - After all, the example dynamic application is already available and what is left is testing it.

Future Work

We plan to make DA-Testing into the final release (1.0.0) within two to three months, hopefully, by then we will have three releases or more. Most of the work will be focused on:

• Supporting OSGi Frameworks other than Equinox, by version 1.0.0 Apache Felix and Knopflerfsih will be supported.
• Allowing developers to create custom conditions.
• Implementing new features and adding new assertions that will further simplify testing dynamic OSGi-based applications.