Step by step Guide¶
1 General¶
sensitivity.io
SDK is written in C++ and provides you with the following modules:
- Sensitive Data Scanner (SDS) with which you can scan strings, byte buffers and files for threats
- Sensitive Data Classification (SDC) with which you can classify strings, byte buffers and files as corporate or public
Each sensitivity.io
module has its own static/shared library:
- sensitivityio_sds
- sensitivityio_sdc
and require the use of the following dependent libraries through which the setup is done:
- sensitivityio_base
- sensitivityio_license
Note
The sensitivity.io
binding libraries provided for the mainstream programming languages respect the same structure.
Important
All SDK components are resources, so a special attention must be taken by devs working in GC languages to close resources. In the pseudocode below we used “using blocks” from C# (similar to try-with-resources from Java, etc)
2 Installation¶
Follow the sensitivity.io
installation guide provided withing the SDK for the required programming language.
3 Module setup¶
Before attempting to use any sensitivity.io
module in your application, you need to first set it up using the following steps:
- initialize resources (where provided)
- load license
- load settings
Resource initialization (where provided) needs to be done only once in the application lifetime before any other operation. Initialize the base resources first, then initialize the specific module resources.
There are 2 modes in which you can perform your license and settings setup, depending on the way you need to handle your license and settings files: offline and online. You can provide both license and settings as either path to those files or string content, depending on how your application implements storage of such resources.
Note
You can have only 1 license and it is unique to your account. The license will be set globally for your application through the defined methods of the singleton LicenseLoader.
3.1 Offline license & settings setup¶
Use offline setup when your license and settings don’t change and you provide them within the application. In this case you need to load your license and settings using the respective loaders.
Log into your sensitivity.io account to download your license and settings files.
To download your license you need to have at least a project and an application created in your account. You select the project from the Projects list, then select the application from the Installations section. You can download your license from the Application Details section.
Load license using LicenseLoader:
LicenseLoader::GetInstance()->LoadLicenseFromFile(licenseFile);
or
LicenseLoader::GetInstance()->LoadLicenseFromString(licenseContent);
You can manage the settings in the Protection Profile section of your sensitivity.io
account. You’ll find you already have some predefined settings, but you can also define your own custom settings based on your own requirements. You can find more on protection profiles by following the link.
Select a protection profile and download its settings to use in your application.
Note
You can have several settings defined in your account, hence, in your application you can have several settings loaded at the same time through different instances of SettingsLoader class or in the same instance of SettingsLoader class but at different times in your app execution.
Use specific module settings loader classes to instantiate a SettingsLoader object:
- ScannerSettingsLoader for SDS module:
using (ScannerSettingsLoader settingsLoader = new ScannerSettingsLoader()) { ... }
- ClassifierSettingsLoader for SDC module:
using (ClassifierSettingsLoader settingsLoader = new ClassifierSettingsLoader()) { ... }
Load module settings:
settingsLoader->LoadSettingsFromFile(settingsFile);
or
settingsLoader->LoadSettingsFromString(settingsContent);
3.2 Online license & settings setup¶
Use online setup when your license and settings change frequently and are downloaded from your sensitivity.io
account using the sensitivity.io
retrievers.
All sensitivity.io
retrievers are subclasses of HttpRetriever and are set up globally through its class methods with your credentials and service details. Setting your credentials is mandatory, if not set, the retrievers will not work. If no service details are set, the default values will be used.
You can find your credentials:
- Auth Key
- Account ID
- Project ID
- Application ID
on the Account Summary panel in your sensitivity.io
account.
Note
When using the RetrieveToFile method of the sensitivity.io
retrievers, you first need to set the destination file for it to work.
Note
When using the RetrieveToFileAndNotifyLoader or RetrieveToStringAndNotifyLoader of the specific module settings retrievers, you need to provide the retriever with the settings loader that will be notified to load the downloaded resource. If the settings loader was not provided then you need to manually load the downloaded resources.
Warning
Retrieving data using the sensitivity.io
retrievers represents a HTTPS request to a sensitivity.io
service and can block the UI if performed on main thread.
Warning
You need to first have a protection profile assigned either globally, to your project or to your application installation specifically for the settings retriever to work properly.
Set up the sensitivity.io
retrievers with your service details through the class methods of HttpRetriever:
HttpRetriever::SetHostName(serviceHostName);
HttpRetriever::SetApiVer(serviceApiVersion);
Note
In case your application uses the default sensivitity.io
service host name and API version (not custom ones) you can skip this step.
Set up the sensitivity.io
retrievers with your credentials through the class methods of HttpRetriever:
HttpRetriever::SetAccountId(yourAccountId);
HttpRetriever::SetProjectId(yourProjectId);
HttpRetriever::SetAuthenticationKey(yourAuthKey);
Retrieve your application ID by using the specialized retriever ApplicationIdRetriever:
ApplicationIdRetriever::GetInstance()->RetrieveApplicationId();
Note
Each application ID is unique to the device your application is installed on, if the same credentials and service details were provided.
Retrieve and load license by using LicenseRetriever singleton class:
LicenseRetriever theLicenseRetriever = LicenseRetriever::GetInstance();
theLicenseRetriever->SetDestinationFile(licenseFilePath);
theLicenseRetriever->RetrieveToFileAndNotifyLoader();
or
LicenseRetriever theLicenseRetriever = LicenseRetriever::GetInstance();
theLicenseRetriever->SetDestinationFile(licenseFilePath);
String license = theLicenseRetriever->RetrieveToStringAndNotifyLoader();
Use specific module settings retriever classes to instantiate a SettingsRetriever object:
- ScannerSettingsRetriever for SDS module:
using (ScannerSettingsRetriever settingsRetriever = new ScannerSettingsRetriever()) { ... }
- ClassifierSettingsRetriever for SDC module:
using (ClassifierSettingsRetriever settingsRetriever = new ClassifierSettingsRetriever()) { ... }
Retrieve settings and notify loader to automatically load the downloaded settings:
settingsRetriever->SetSettingsLoader(settingsLoader);
settingsRetriever->SetDestinationFile(settingsFilePath);
settingsRetriever->RetrieveToFileAndNotifyLoader();
or
settingsRetriever->SetSettingsLoader(settingsLoader);
String settings = settingsRetriever->RetrieveToStringAndNotifyLoader();
Clear the settings loader when you are finished with the retriever that has set the settings loader to automatically load the downloaded settings:
settingsRetriever->SetSettingsLoader(nil);
You can also set an auto retrieve interval for the settings, in case they change frequently, thus they will be downloaded and loaded at the specified interval. The destination file needs to be set for the auto retrieve functionality to work properly, otherwise the retrieval will fail until a destination file is set.
settingsRetriever->SetDestinationFile(settingsFile);
settingsRetriever->SetAutoRetrieveInterval(customAutoRetriveInterval);
Warning
The auto retrieve interval functionality works only in Qt based applications and in the bindings that have this functionality custom implemented.
4 Sensitive Data Scanner (SDS) module¶
Sensitive Data Scanner module provides the means for scanning data locally, with no need for internet connection.
There are two ways you can perform a scan:
- by using a single thread scanner, which is represented by Scanner class, or
- by using a multi thread scanner, which is represented by ScannerExecutor class
So choose whichever fits your requirements best.
Before performing any scans the following steps are required:
- the module needs to have its resources initialized (where provided)
- the license needs to be loaded
- the scanner settings need to be loaded
- the scanner needs to be registered with the settings loader containing the loaded settings
Note
When registering a scanner with a scanner settings loader, the already loaded settings will propagate to the new registered scanner. Whenever new settings are loaded with a settings loader that has registered scanners, the new settings will be propagated to the registered scanners.
Warning
Always unregister scanner from settings loader when you no longer need it.
Each type of scanner has 2 types of scan methods:
convenience scan where you either:
- perform a full scan
- stop at the first encountered threat based on the boolean value you submit for the stopAtFirstThreat parameter
- stop at maxNumThreats based on the integer value you submit for the maxNumThreats parameter
scan with threat handler where you define your own stop condition by handling yourself each found threat through your own threat handler
The structure of a threat object is represented by ThreatInfo class. It has the following properties:
- typeId which represents the threat id
- typeName which represents the threat name
- matchedText which represents the text that is considered as threat
- maskedMatchedText which represents a masked version of the text that is considered as threat
- threatDescription which represents the description of the threat
- location which represents the location of the threat
The structure of a threat location is defined through ThreatLocation class and has the following properties:
- offset which represents the offset where the threat was found, -1 means this information is not available
- lineNumber which represents the line number where threat was found, 0 means this information is not available
- fileName which represents the name of the file that was scanned. Non null only when scanning files
- archiveMember which represents the name of the file where the threat was found inside the scanned archive. Missing if fileName is not an archive
- surroundingText which represents the surrounding text of the threat
- maskedSurroundingText which represents a masked version of the surrounding text of the threat
- inMetadata which represents the location of the found threat as being in metadata or not
Use ThreatHandler class to define your own threat handler:
class TestThreatHandler: ThreatHandler {
public TestThreatHandler(int maxNumThreats) {
numThreats_ = 0;
kMaxNumThreats_ = maxNumThreats;
}
public override bool HandleThreat(ThreatInfo threatInfo) {
Print("Threat #", numThreats_, ": ", threatInfo->ToString());
++numThreats_;
return (kMaxNumThreats_ == 0) ? true : ((numThreats_ == kMaxNumThreats_) ? false : true);
}
private int numThreats_;
private int kMaxNumThreats_;
};
4.1 Single thread scanner¶
Use Scanner class to scan strings, byte buffers and files one at a time.
Note
When cancelling a scanner, it will become invalid, thus no more scans can be performed with it. In order to perform a new scan you need to create a new instance of Scanner.
Warning
Cancelling a scanner does not immediately stop the ongoing scan, so, before releasing a scanner, you need to make sure that the scan finished to avoid memory leaks.
Initialize the scanner:
using (Scanner scanner = new Scanner()) { ... }
Register the scanner with the specific settings loader for the SDS module, ScannerSettingsLoader:
settingsLoader->RegisterScanner(scanner);
Whole file convenience scan:
bool stopAtFirstThreat = false;
using (ThreatInfos threatInfos = scanner->ScanFile(fileToScan, stopAtFirstThreat)) { ... }
or
int maxNumThreats = 0;
using (ThreatInfos threatInfos = scanner->ScanFile(fileToScan, maxNumThreats)) { ... }
Convenience scan file and stop at first encountered threat:
bool stopAtFirstThreat = true;
using (ThreatInfos threatInfos = scanner->ScanFile(fileToScan, stopAtFirstThreat)) { ... }
or
int maxNumThreats = 1;
using (ThreatInfos threatInfos = scanner->ScanFile(fileToScan, maxNumThreats)) { ... }
Whole file scan with previously defined test threat handler:
TestThreatHandler testThreatHandler = TestThreatHandler(0);
scanner->ScanFile(fileToScan, testThreatHandler);
Scan file with previously defined test threat handler and stop at first encountered threat:
TestThreatHandler testThreatHandler = TestThreatHandler(1);
scanner->ScanFile(fileToScan, testThreatHandler);
4.2 Multi thread scanner¶
Use ScannerExecutor class to scan strings, byte buffers and files simultaneously through an internal thread pool.
Use ThreadPool class to monitor and configure the scanner executor internal thread pool:
- get and set the maximum number of threads used by the thread pool
- get the number of active threads existing in the thread pool
- get and set threads expiry timeout for the thread pool. The default value is 30000 milliseconds
- wait until all threads exit and are removed from the thread pool, with the possibility to set a timeout for the wait. Will block your UI if done on main thread
Each scan method of the scanner executor returns a “Future” object similar to the Qt QFuture concept, through which you can manipulate and check the state of the scan.
Depending on the type of scan:
- convenience scan: the returned future object is of type ThreatInfosFuture
- scan with threat handler: the returned future object is of type VoidFuture
The scan methods require you to specify a priority for each scan. High priority scans execute before low priority ones.
Warning
Before releasing a scanner executor you need to make sure that its corresponding thread pool is empty in order to avoid blocking your execution thread in the destructor till all remaining tasks finish.
Initialize the scanner executor:
using (ScannerExecutor scannerExecutor = new ScannerExecutor()) { ... }
Register the scanner executor with the specific settings loader for the SDS module, ScannerSettingsLoader:
settingsLoader->RegisterScannerExecutor(scannerExecutor);
Whole file convenience scan:
bool stopAtFirstThreat = false;
using (ThreatInfosFuture threatInfosFuture = scannerExecutor->ScanFile(fileToScan, stopAtFirstThreat, priority)) {
using (ThreatInfos threats = threatInfosFuture->GetResult()) { ... }
}
or
int maxNumThreats = 0;
using (ThreatInfosFuture threatInfosFuture = scannerExecutor->ScanFile(fileToScan, maxNumThreats, priority)) {
using (ThreatInfos threats = threatInfosFuture->GetResult()) { ... }
}
Convenience scan file and stop at first encountered threat:
bool stopAtFirstThreat = true;
using (ThreatInfosFuture threatInfosFuture = scannerExecutor->ScanFile(fileToScan, stopAtFirstThreat, priority)) {
using (ThreatInfos threats = threatInfosFuture->GetResult()) { ... }
}
or
int maxNumThreats = 1;
using (ThreatInfosFuture threatInfosFuture = scannerExecutor->ScanFile(fileToScan, maxNumThreats, priority)) {
using (ThreatInfos threats = threatInfosFuture->GetResult()) { ... }
}
Whole file scan with previously defined test threat handler:
TestThreatHandler testThreatHandler = TestThreatHandler(0);
using (VoidFuture voidFuture = scannerExecutor->ScanFile(fileToScan, testThreatHandler, priority)) {
voidFuture->WaitForFinished();
}
Scan file with previously defined test threat handler and stop at first encountered threat:
TestThreatHandler testThreatHandler = TestThreatHandler(1);
using (VoidFuture voidFuture = scannerExecutor->ScanFile(fileToScan, testThreatHandler, priority)) {
voidFuture->WaitForFinished();
}
Cancelling scanner executor pending and ongoing scans is done through the “Future” objects by requesting a task cancellation:
future->Cancel()
Warning
This does not mean that the future underlying task finished cancelling. The task finished cancelling when WaitForFinished fails with canceled error/exception.
Before releasing the scanner executor, wait for thread pool to empty:
scannerExecutor->ThreadPool()->WaitForFinished()
Warning
Keep in mind that the above method will block and wait until all threads exit and are removed from the thread pool.
4.3 Whole Sample¶
main() {
try {
// TODO: modify as desired
const bool kEnableOnline = false;
const String kAccountId = your_account_id;
const String kProjectId = your_project_id;
const String kAuthenticationKey = your_authentication_key;
const String kApplicationId = ""; // fill if known
const String kLicenseFile = kEnableOnline ? writable_download_path : existing_license_path;
const String kSettingsFile = kEnableOnline ? writable_download_path : existing_settings_path;
const String kFileToScan = file_to_scan;
if (LogToFile::IsLogToFileEnabled) {
String logFileName = LogToFile::GetLogFileName();
if (LogToFile::InstallLogFileMsgHandler(logFileName)) {
Print("Logs will be placed in file: ", logFileName);
}
else {
Print("Failed to setup logging to file: ", logFileName);
}
}
Print("sensitivityio version: ", Version::ToString());
Print("Test starting...");
if (kEnableOnline) {
Print("Application Id...");
HttpRetriever::SetAccountId(kAccountId);
HttpRetriever::SetProjectId(kProjectId);
HttpRetriever::SetAuthenticationKey(kAuthenticationKey);
HttpRetriever::SetApplicationId(kApplicationId);
if (HttpRetriever::GetApplicationId() == String.Empty) {
ApplicationIdRetriever::GetInstance()->RetrieveApplicationId();
Print("Retrieved application Id: ", HttpRetriever::GetApplicationId());
}
}
Print("License...");
if (kEnableOnline) {
LicenseRetriever theLicenseRetriever = LicenseRetriever::GetInstance();
theLicenseRetriever->SetDestinationFile(kLicenseFile);
theLicenseRetriever->RetrieveToFileAndNotifyLoader();
}
else {
LicenseLoader::GetInstance()->LoadLicenseFromFile(kLicenseFile);
}
Print("Settings...");
using (ScannerSettingsLoader scannerSettingsLoader = new ScannerSettingsLoader()) {
if (kEnableOnline) {
using (ScannerSettingsRetriever scannerSettingsRetriever = new ScannerSettingsRetriever()) {
scannerSettingsRetriever->SetDestinationFile(kSettingsFile);
scannerSettingsRetriever->SetSettingsLoader(scannerSettingsLoader);
try {
scannerSettingsRetriever->RetrieveToFileAndNotifyLoader();
}
finally {
scannerSettingsRetriever->SetSettingsLoader(null);
}
}
}
else {
scannerSettingsLoader->LoadSettingsFromFile(kSettingsFile);
}
Print("Scanning...");
using (ScannerExecutor scannerExecutor = new ScannerExecutor()) {
scannerSettingsLoader->RegisterScannerExecutor(scannerExecutor);
try {
if (kEnableTestThreatHandler) {
TestThreatHandler testThreatHandler = new TestThreatHandler(kMaxNumThreats);
using (VoidFuture voidFuture = scannerExecutor->ScanFile(kFileToScan, testThreatHandler, 0)) {
voidFuture->WaitForFinished();
}
}
else {
using (ThreatInfosFuture threatInfosFuture = scannerExecutor->ScanFile(kFileToScan, kMaxNumThreats, 0)) {
using (ThreatInfos threatInfos = threatInfosFuture->GetResult()) {
int threatInfosSize = threatInfos->GetSize();
Print("Found ", threatInfosSize, " threats:");
for (int threatNum = 0; threatNum < threatInfosSize; ++threatNum) {
using (ThreatInfo threat = threatInfos->GetElement()) {
Print("Threat #", threatNum, ": ", threat->ToString());
}
}
}
}
}
}
finally {
scannerSettingsLoader->UnregisterScannerExecutor(scannerExecutor);
}
}
}
Print("Test completed");
}
catch (Exception exc) {
Print("Test failed: ", exc->ToString());
}
}
5 Sensitive Data Classification (SDC) module¶
Sensitive Data Classification module provides the means for classifying data locally, with no need for internet connection, into corporate or public.
There are two ways you can perform a classification:
- by using a single thread classifier, which is represented by Classifier class, or
- by using a multi thread classifier, which is represented by ClassifierExecutor class
So choose whichever fits your requirements best.
Before performing any classifications, the following steps are required:
- the module needs to have its resources initialized (where provided)
- the license needs to be loaded
- the classifier settings need to be loaded
- the classifier needs to be registered with the settings loader containing the loaded settings
Note
When registering a classifier with a classifier settings loader, the already loaded settings will propagate to the new registered classifier. Whenever new settings are loaded with a settings loader that has registered classifiers, the new settings will be propagated to the registered classifiers.
Warning
Always unregister classifiers from the settings loader when you no longer need them.
Both Classifier and ClassifierExecutor provide you methods to classify strings, byte buffers and files and result in a Classification object that states if the provided data for classification is either corporate or public.
5.1 Single thread classifier¶
Use Classifier class to classify strings, byte buffers and files one at a time.
Note
When cancelling a classifier, it will become invalid, thus no more classifications can be performed with it. In order to perform a new classification you need to create a new instance of Classifier.
Warning
Cancelling a classifier does not immediately stop the ongoing classification, so, before releasing a classifier, you need to make sure that the classification finished to avoid memory leaks.
Initialize the classifier object:
using (Classifier classifier = new Classifier()) { ... }
Register the classifier with the specific settings loader for the SDC module, ClassifierSettingsLoader:
settingsLoader->RegisterClassifier(classifier)
Classify file:
using (Classification classification = classifier->ClassifyFile(fileToClassify)) { ... }
5.2 Multi thread classifier¶
Use ClassifierExecutor class to classify strings, byte buffers and files simultaneously through an internal thread pool.
Use ThreadPool class to monitor and configure the classifier executor thread pool:
- get and set the maximum number of threads used by the thread pool
- get the number of active threads existing in the thread pool
- get and set threads expiry timeout for that thread pool. The default value is 30000 milliseconds
- wait untill all threads exit and are removed from the thread pool, with the posibility to set a timeout for the wait. Will block your UI if done on main thread
Each classify method of the classifier executor returns a ClassificationFuture object similar to the Qt QFuture concept, through which you can manipulate and check the state of the classification.
The classify methods require you to specify a priority. High priority classifications execute before low priority ones.
Note
Before releasing a classifier executor you need to make sure that its corresponding thread pool is empty in order to avoid blocking your execution thread in the destructor till all remaining tasks finish.
Initialize the classifier executor:
using (ClassifierExecutor classifierExecutor = new ClassifierExecutor()) { ... }
Register the classifier with the specific settings loader for the SDS module, ClassifierSettingsLoader:
settingsLoader->RegisterClassifierExecutor(classifierExecutor)
Classify file:
using (ClassificationFuture classificationFuture = classifierExecutor.ClassifyFile(fileToClassify, 0)) {
using (Classification classification = classificationFuture.GetResult()) { ... }
}
Cancelling classifier executor pending and ongoing classifications is done through the “Future” objects by requesting a task cancellation:
future->Cancel()
Warning
This does not mean that the future underlying task finished cancelling. The task finished cancelling when WaitForFinished fails with canceled error/exception.
Before releasing the classifier executor, wait for thread pool to empty:
classifierExecutor->ThreadPool()->WaitForFinished()
Warning
Keep in mind that the above method will block and wait until all threads exit and are removed from the thread pool.
5.3 Whole Sample¶
main() {
try {
// TODO: modify as desired
const bool kEnableOnline = false;
const String kAccountId = your_account_id;
const String kProjectId = your_project_id;
const String kAuthenticationKey = your_authentication_key;
const String kApplicationId = ""; // fill if known
const String kLicenseFile = kEnableOnline ? writable_download_path : existing_license_path;
const String kSettingsFile = kEnableOnline ? writable_download_path : existing_settings_path;
const String kFileToClassify = file_to_classify;
if (LogToFile::IsLogToFileEnabled) {
String logFileName = LogToFile::GetLogFileName();
if (LogToFile::InstallLogFileMsgHandler(logFileName)) {
Print("Logs will be placed in file: ", logFileName);
}
else {
Print("Failed to setup logging to file: ", logFileName);
}
}
Print("sensitivityio version: ", Version::ToString());
Print("Test starting...");
if (kEnableOnline) {
Print("Application Id...");
HttpRetriever::SetAccountId(kAccountId);
HttpRetriever::SetProjectId(kProjectId);
HttpRetriever::SetAuthenticationKey(kAuthenticationKey);
HttpRetriever::SetApplicationId(kApplicationId);
if (HttpRetriever::GetApplicationId() == String.Empty) {
ApplicationIdRetriever::GetInstance()->RetrieveApplicationId();
Print("Retrieved application Id: ", HttpRetriever::GetApplicationId());
}
}
Print("License...");
if (kEnableOnline) {
LicenseRetriever theLicenseRetriever = LicenseRetriever::GetInstance();
theLicenseRetriever->SetDestinationFile(kLicenseFile);
theLicenseRetriever->RetrieveToFileAndNotifyLoader();
}
else {
LicenseLoader::GetInstance()->LoadLicenseFromFile(kLicenseFile);
}
Print("Settings...");
using (ClassifierSettingsLoader classifierSettingsLoader = new ClassifierSettingsLoader()) {
if (kEnableOnline) {
using (ClassifierSettingsRetriever classifierSettingsRetriever = new ClassifierSettingsRetriever()) {
classifierSettingsRetriever->SetDestinationFile(kSettingsFile);
classifierSettingsRetriever->SetSettingsLoader(classifierSettingsLoader);
try {
classifierSettingsRetriever->RetrieveToFileAndNotifyLoader();
}
finally {
classifierSettingsRetriever->SetSettingsLoader(null);
}
}
}
else {
classifierSettingsLoader->LoadSettingsFromFile(kSettingsFile);
}
Print("Classifying...");
using (ClassifierExecutor classifierExecutor = new ClassifierExecutor()) {
classifierSettingsLoader->RegisterClassifierExecutor(classifierExecutor);
try {
using (ClassificationFuture classificationFuture = classifierExecutor->ClassifyFile(kFileToClassify, 0)) {
using (Classification classification = classificationFuture->GetResult()) {
Print("classification: ", classification->ToString());
}
}
}
finally {
classifierSettingsLoader->UnregisterClassifierExecutor(classifierExecutor);
}
}
}
Print("Test completed");
}
catch (Exception exc) {
Print("Test failed: ", exc->ToString());
}
}