Usage

	a_column_identifier
	"a quoted column identifier"
						

Column Directives

There are four OPTIONAL Column Directives that are used to modify aspects of how the Column Rules are evaluated. Like Global Directives, Column Directives are indicated by the Directive Prefix, defined as the character @ i.e. the [[UTF-8]] character code 0x40.

The Column Directives are the Optional Directive, the Match Is False Directive, the Ignore Case Directive, and the Warning Directive. The column directives may be specified in any order (though there is no straightforward way to express this in EBNF without listing every possible order).

Explicit Context Expressions

The Explicit Context Expression is used to indicate that the expression following should be tested against the value in a foreign column (explicit context), rather than the current column (which is the default context).

	a_column: is("some string") and $another_column/starts("some string") //here two tests are combined on a single line, the second test here looks to the second column 
	another_column:                                                       //to check it's value starts with "some string"
							

Is Expressions

An Is Expression checks that the value of the column is identical to the supplied string or the value in the referenced column.

	a_column: is("some string")    //the contents of a_column must be the string "some string"
	another_column: is($a_column)  //the contents of another_column must be the value held in a_column, treated as a string
							

Any Expressions

This is a new expression in CSV Schema Language 1.1

An Any Expression checks that the value of the column is identical to one of the supplied strings or the values in the referenced columns. This is logically equivalent to a series of Is Expressions joined by Or Expressions, but slightly more compact to write and maintain.

	a_column: is("some string")
	another_column: any("some other string",$a_column,"another string") //any of the string values given here are valid, including referencing the string held in a_column
							

Not Expressions

A Not Expression checks that the value of the column is not equal to the supplied string or the value in the referenced column.

	a_column: not("some string")     //the value of a_column must not be the string "some string"
	another_column: not($a_column)   //the value of another_column must not be the value held in a_column
							

In Expressions

An In Expression checks that the value of the column contains the supplied string or the value in the referenced column (i.e. the column value is a super string of the supplied value).

	a_column: in("some string")   //the value of a_column must be a substring of "some string" eg "some" or "string" or "me st" etc
	another_column: in($a_column) //the value of another_column must be a substring of the contents of a_column
							

Starts With Expressions

A Starts With Expression checks that the value of the column starts with the supplied string or the value in the referenced column.

	a_column: starts("some string")   //the value of a_column must start with the string "some string" eg "some strings" or "some string is here that's really long"
	another_column: starts($a_column) //the value of another_column must start with the contents of a_column
							

Ends With Expressions

An Ends With Expression checks that the value of the column ends with the supplied string or the value in the referenced column.

	a_column: ends("some string")   //the value of a_column must end with the string "some string" eg "here is some string" or "this really long string ends with some string"
	another_column: ends($a_column) //the value of another_column must end with the contents of a_column
							

Regular Expression Expressions

A Regular Expression Expression checks the value of the column against the supplied Regular Expression.

Whilst obviously many of the the other Column Validation Expressions could be written as Regular Expressions, it is felt that that would make the resulting Schema harder to read, and much harder to write for less technical users. As such it is recommended, that if a Column Rule can be written without regular expressions, by instead using other Column Validation Expressions, then that approach should be attempted first.

The Regular Expression syntax used is that from Java's Pattern class.

	a_column: regex("[bcm]at")   //the value of a_column must match the regular expression [bcm]at ie a string containing "bat", "cat" or "mat"
	another_column: regex("[0-5]") //the value of another_column match the regular expression [0-5] ie a string containing only the digits only 0-5.
						

Range Expressions

The definition of this expression in CSV Schema Language 1.1 extends the definition originally made in CSV Schema Language 1.0

A Range Expression checks that the value of the column is a number lying between, or equal to, the supplied upper and lower bounds. One or other bounding value MAY be left unrestricted (by using the Wildcard Literal), so that the expression can also be used to check that a column is at least some value, or at most some value. One bounding value is defined as a Numeric Or Any expression which accepts a Numeric Literal (a Real Number expressed as a decimal), or the Wildcard Literal, while the other is a Numeric Literal only. Therefore at least one of the bounding values MUST be a Numeric Literal rather than a Wildcard Literal, so valid Range Expressions SHALL define ranges of:

1. At least n. For example range(10, *) states that the column value MUST be at least 10
2. At most n. For example range(*, 10) ensures that the column value MUST be at most 10 (and may be negative).
3. Between m and n(inclusive). For example range(4, 7) ensures that the column value MUST be between 4 and 7 (inclusive).

Length Expressions

A Length Expression checks that the number of characters in the column meets the supplied definition. You can define the length in one of four ways:

1. Precisely n characters long. For example length(10) ensures that the length is exactly 10 characters.
2. At least n characters long. For example length(10, *) ensures that the length is at least 10 or more characters
3. At most n characters long. For example length(*, 10) ensures that the length is at most 10 characters.
4. Between m and n characters long (inclusive). For example length(4, 7) ensures that the length is at least 4 characters and at most 7 characters.

In order to do this the expression takes two input parameters, the first is defined as OPTIONAL, the second MUST be supplied. Both take the form of a Positive Integer Or Any expression, which is either a Positive Integer Literal (which actually includes zero), or a Wildcard Literal.

Empty Expressions

An Empty Expression checks that the column has no content.

	a_column: empty //there must be no value in a_column
							

Not Empty Expression

A Not Empty Expression checks that the column has some content, though precisely what does not matter.

	a_column: notEmpty   //there must be some value in a_column, but it can be absolutely anything
							

Unique Expressions

A Unique Expression checks that the column value is unique within the CSV file being validated (within the current column, the value may occur elsewhere in the file in another column, as in a primary-foreign key relationship in a database). You can also specify a comma separated list of Column References in which case the combination of values of those columns (for the current row) must be unique within the whole CSV file.

Usage

	a_column: unique                                  //a_column must hold a unique value in each row of the CSV file
	another_column: unique($a_column,$another_column) //the combination of values in the two columns must be unique looking across each row of the CSV file
							

URI Expressions

A URI Expression means that the value in the column MSUT be a valid URI as defined in [[!RFC3986]].

	a_column: uri   //the value of a_column must be a valid URI
							

XSD Date Time Expressions

An XSD Date Time Expression checks that the data in the column is expressed as a valid XML Schema dateTime data type (see [[!XMLSCHEMA-2]] and [[!ISO8601]]). You can also provide an OPTIONAL from and to date-times (inclusive) to ensure that the value in the column falls within an expected date-time range.

As the XSD Date Time Expression uses the XSD Date Time Literal the final, timezone, part of the [[!ISO8601]] definition is OPTIONAL.

	a_column: xDateTime                                                 //the value of a_column must be a valid xDateTime
	another_column: xDateTime(2014-10-04T00:00:01Z,2015-12-03T23:59:59) //the value of another_column must be a valid xDateTime and between the two xDateTimes shown (inclusive)
	                                                                    //as shown, the xDateTime values may, or may not, have a component indicating a specific timezone, here Z (Zulu) for UTC (Greenwich Mean Time)
							

XSD Date Time With Time Zone Expressions

An XSD Date Time With Time Zone Expression is identical to XSD Date Time Expression except that it uses XSD Date Time With Time Zone Literal rather than XSD Time Literal, which means that the time zone component in [[!XMLSCHEMA-2]] and [[!ISO8601]] MUST be used. Again, you can also provide an OPTIONAL from and to date-times, (inclusive) to ensure that the value in the column falls within an expected date-time range. These are also defined as XSD Date Time With Time Zone Literal so MUST have the time zone component.

	a_column: xDateTime                                                            //the value of a_column must be a valid xDateTime
	another_column: xDateTime(2014-10-04T00:00:01+02:00,2015-12-03T23:59:59+02:00) //the value of another_column must be a valid xDateTime and between the two xDateTimes shown (inclusive)
	                                                                               //now the time zone component (+02:00) must be included
							

XSD Date Expressions

An XSD Date Expression checks that the data in the column is expressed as a valid XML Schema date data type (see [[!XMLSCHEMA-2]] and [[!ISO8601]]). You can also provide OPTIONAL from and to dates (inclusive) to ensure that the value in the column falls within an expected date range.

	a_column: xDate                                  //the value of a_column must be a valid xDate
	another_column: xDateTime(2014-10-04,2015-12-03) //the value of another_column must be a valid xDate and between the two xDates shown (inclusive)
							

XSD Time Expressions

An XSD Time Expression checks that the data in the column is expressed as a valid XML Schema time data type (see [[!XMLSCHEMA-2]] and [[!ISO8601]]). You can also provide OPTIONAL from and to times (inclusive) to ensure that the value in the column falls within an expected time range.

	a_column: xTime                                                 //the value of a_column must be a valid xTime
	another_column: xTime(00:00:01+02:00,2015-12-03T23:59:59+02:00) //the value of another_column must be a valid xTime and between the two xTimes shown (inclusive)
	                                                                //the time zone component (+02:00) is optional
							

UK Date Expressions

A UK Date Expression checks that the data in the column is expressed as a valid UK-style date: dd/mm/yyyy, a UK Date Literal. You can also provide OPTIONAL from and to dates (inclusive) to ensure that the value in the column falls within an expected date range.

	a_column: ukDate                              //the value of a_column must be a valid ukDate
	another_column: ukDate(04/10/2014,03/12/2015) //the value of another_column must be a valid ukDate and between the two ukDates shown (inclusive)
							

Date Expression

A Date Expression allows a full date to be constructed from several columns, or strings. The expression takes five arguments: you MUST supply three strings or Column References representing Year, Month and Day (although supplied as strings, these values must in fact be integers); and there are two OPTIONAL parameters to ensure the date falls in a range specified by from XSD Date Expression and to XSD Date Expression.

	year_column: date($year_column,$month_column,$day_column)                      /*the full date to be checked is made up from the values from the three columns indicated, 
	                                                                                 as shown, the columns can appear in CSV file in any order, but must appear in the date expression in the order year, month, day*/
	day_column: date($year_column,$month_column,$day_column,2014-10-04,2015-12-03) //this second example shows that you can also supply from and to dates as xDates, both dates are inclusive
	month_column:
							

Partial UK Date Expression

A Partial UK Date Expression is essentially the same as a UK Date Expression, but allows for difficulties in transcribing from original archival material by accepting a question mark (?), i.e. the [[UTF-8]] character code 0x3F in place of illegible digits in any position, or an asterisk (*), i.e. the [[UTF-8]] character code 0x2A in place of a missing value. The names of months may also be supplied as the full name in English, i.e.: January, February, March, April, May, June, July, September, October, November, December. As dates may not be complete, it is impossible to determine reliably if a date falls within a given range, so there is no option to supply one.

	a_column: partUkDate //the value of a_column must be a valid ukDate, but may also include characters ? and * to represent illegible or missing characters, and month names in full
							

Partial Date Expression

A Partial Date Expression combines elements of Partial UK Date Expression with those of Date Expression, with the date being made up of columns or strings as in Date Expression, but also allowing the characters representing uncertainty as in Partial UK Date Expression. However, the constituent parts of the date MUST be supplied as Year, Month, Day.

	year_column: date($year_column,$month_column,$day_column)                      //the full date to be checked is made up from the values from the three columns indicated, 
	day_column:                                                                    //as shown, the columns can appear in CSV file in any order, but must appear in the date expression in the order year, month, day
	month_column:                                                                  //the date may also include characters ? and * to represent illegible or missing characters
							

UUID4 Expression

A UUID4 Expression checks that the data in the column is in the form of a Version 4 UUID (Universally Unique Identifier), see [[!RFC4122]]. UUIDs MUST use lowercase hex values.

	a_column: uuid4 //the value of a_column must be a valid version 4 UUID
							

Positive Integer Expression

A Positive Integer Expression checks that the column contains an integer value, greater than or equal to zero.

	a_column: positiveInteger //the value of a_column must be a positive integer (including zero)
							

Upper Case Expression

This is a new expression in CSV Schema Language 1.1

An Upper Case Expression checks that the column content is all upper case, for all code points in the [[!UTF-8]] character set which have a defined case. Non-cased code points (e.g. numeric characters, punctuation and white space) are also permitted.

In Java's Pattern class this could be expressed via the following Regular Expression: "^[\p{Lu}\p{N}\p{P}\s]*$".

	a_column: upperCase //the contents of a_column must be all be upper case characters (Unicode aware), or uncased
							

Lower Case Expression

This is a new expression in CSV Schema Language 1.1

A Lower Case Expression checks that the column content is all lower case, for all code points in the [[!UTF-8]] character set which have a defined case. Non-cased code points (e.g. numeric characters, punctuation and white space) are also permitted.

In Java's Pattern class this could be expressed via the following Regular Expression: "^[\p{Ll}\p{N}\p{P}\s]*$".

	a_column: lowerCase //the contents of a_column must be all be lower case characters (Unicode aware), or uncased
							

Identical Expressions

This is a new expression in CSV Schema Language 1.1

An Identical Expression asserts that the value of the column MUST be identical for every row within a CSV file, without having to specify precisely what that value will be when writing the CSV Schema.

	batch_date:	identical xDate
	batch_code:	identical regex("^[A-Z]{3,5}B[0-9]{3}$")
								

File Exists Expressions

A File Exists Expression checks a filesystem to see if the specified file actually exists at the specified path. It takes an OPTIONAL expression in the form of a String Provider which allows you to supply a string (or reference to a string) with a full filepath to prepend to the contents of the current column (in the case that for example it only contains just the name of the file). The default value for this string is an empty string.

See also the Integrity Check Expression which performs the inverse function of ensuring that all files in a given folder structure have been mentioned in a CSV file.

	a_column: fileExists                      //the validator should check the filesystem location indicated by the contents of a_column for the existence of such a file
	another_column: fileExists("file:///C:/") //here the string "file:///C:/" is prepended to the contents of another_column before the existence check is made
							

Integrity Check Expressions

This is a new expression in CSV Schema Language 1.1

An Integrity Check Expression checks a filesystem to see if there are any files present that are not specifically mentioned in the CSV file. It takes three expressions as parameters. The first are two OPTIONAL expressions in the form of String Providers. The first (as in the File Exists Expression) allows you to supply a string (or reference to a string) with a full filepath to prepend to the contents of the current column (in the case that for example it only contains just the name of the file). The default value for this string is an empty string. The second parameter allows a string provider to be given to point to an explicit subdirectory relative to the location of the base path. By default this subdirectory is expected to be called "content". If only a single OPTIONAL parameter is supplied, it will be assumed to be the first, so if you wish to set only the second OPTIONAL parameter, you MUST also explicitly supply the first as an empty string ("") The final expression MUST be supplied. This indicates whether references to subfolders are explicitly included in the CSV file, if the CSV file has a row for each subfolder the exact string "includeFolder" should be given, if the subfolders do not have explicit references, the exact string "excludeFolder" should be given.

Default treatment of case sensitivity should follow the norms of the relevant file system, implementations may wish to include some means to over-ride this, but that is outside the scope of the EBNF.

	a_column: integrityCheck("includeFolder")                     /*the validator should check all file system folders for files that are not referenced in the CSV file
	                                                                the "includeFolder" parameter indicates that there are explicit references to all file folders in the CSV file
																    as the second parameter has not been supplied it defaults to the value "content" meaning that all sub folders must
																	sit within a folder with that name*/
	another_column: integrityCheck("file:///C:/","excludeFolder") //here the string "file:///C:/" is prepended to the contents of another_column before the integrity check is made
	third_column: integrityCheck("","","excludeFolder")           //here as an strings are passed for both optional parameters, we indicate that there is no content folder
							

Checksum Expressions

A Checksum Expression allows the verification of a checksum value supplied in a CSV file by calculating the checksum for a specific file, and comparing it against the supplied value. You can also indicate the type of checksum algorithm to be used. You MUST supply both the file location and a checksum algorithm. File location is given in the form of a File Expression.

The EBNF does not specify valid values for the String Literal representing the checksum algorithm, that is instead implementation defined, however it is strongly recommended that implementations SHOULD at least support: MD5, SHA-1 and SHA-256. It is important to note that the checksum value MUST use lowercase hexadecimal characters only.

	file_path: uri                                                           //a full filepath for a file
	file_name:                                                               //a filename only for a file
	a_column: checksum(file($file_path),"SHA-256")                           /*a_column contains the SHA-256 checksum value supplied for the file, the validator
                                                                               then computes the checksum for the file at file_path and reports an error if they do not match*/
	another_column: checksum(file("file:///C:/my_folder/",$file_name),"MD5") /*here the string "file:///C:/my_folder" is prepended to the contents of file_name before the 
	                                                                           checksum verification is performed, another_column has an MD5 checksum value in it
							

File Count Expressions

A File Count Expression allows a column representing the number of files in a particular folder to be verified against the actual files on disk. You MUST provide a File Expression as an input parameter which points to a filesystem folder to compare the count of files against.

	file_path: uri                                      //a full filepath for a folder
	another_column: positiveInteger fileCount(file($file_path)) //another_column contains an integer value, this is checked against the number of files in the file system folder at file_path
							

File related sub-expressions

The sub-expression used in conjunction with External Single Expressions in order to provide input is a generic File Expression which itself takes two parameters. The first parameter is OPTIONAL, a String Provider that is prepended to the second parameter to create a full path in the event that a column holds only a filename rather than a full filepath. You MUST supply the second parameter which is a String Provider, which resolves to the name of the file.

	file("file:///a/string/for/a/URI/representing/the/location/of/a/file")           /*in the simplest case a single parameter is supplied that's a string representing the full URI of a
	                                                                                   file. Since it's a URI, characters such as space must be escaped (a space becomes %20)
																			           you can either use a string literal as here, or pass a reference to a column using $column_name*/
	file("file:///a/string/to/prepend/to/a/filename/to/make/a/full/path","filename") /*Here you provide a string (or reference to a string) that is the base path to prepend to "filename"
	                                                                                   to get the full path to your file*/
							

Usage

	a_column: is("no file")                                     
	another_column: any("file:///","https://")
	third_column: any("C:","example.com")
	fourth_column: ends(".html")
	fifth_column: is($a_column) or is(concat($another_column,$third_column,"/",noExt($fourth_column),".pdf")
	/*in this rather artificial example, fifth_column must be either "no file" (the value of a_column) or a PDF file with the same basic name as the HTML file named in fourth_column,
	  located at either file:///C:/ or https://example.com/ (in fact as written you could have file:///example.com/ or https://C:/ as well)*/
						

Usage

	a_column: any("true","false")
	another_column: any("yes","no")
	third_column: if($a_column\is("true"),is("some string"),is("some other string")) /*here we look to the value of a_column, if it is "true", third_column must be "some string"
	                                                                                   otherwise third_column must be "some other string"*/
	fourth_column: if(($a_column\is("true") and $another_column\is("yes")),is("some string"),is("some other string"))
	//in fourth_column we use a more complicated combinatorial expression as the initial test, if both a_column is "true" and another_column is "yes", 
	//then fourth_column is "some string", otherwise fourth_column is "some other string".  All column expressions could be used for the test etc, only is is used for simpilcity
						

Switch Case Expression

The Switch Case Expression takes two parameters (equivalent to the first two non-optional parameters of the If Expression), first a Combinatorial Expression or Non Conditional Expression; when that evaluates to true, the second parameter, one or more Column Validation Expressions, are applied. At least one Switch Case Expression MUST be used within a Switch Expression, but there is no limit on the maximum number used.

Usage

	a_column: any("true","false","unknown")
	another_column: if($a_column\is("true"),is("some string"),if($a_column\is("false"),is("some other string"),is("some third string")))
	/*here we look to the value of a_column, if it is "true", another_column must be "some string" otherwise, check if a_column is "false), 
	if so another_column must be "some other string", otherwise another_column is "some third string".  Nesting if statments like this can quickly get 
	very difficult to read, so instead we can use the switch statement*/
	third_column: switch(($a_column\is("true"),is("some string")),($a_column\is("false"),is("some other string")),is("some third string"))
	//this switch statement for third_column is functionally identical to the nested if statement demonstrated on another_column, and is much easier to extend
	//if there were additional options available for a_column, each bracketed pair such of test and column validation expression, 
	//such as ($a_column\is("true"),is("some string")), is called a Switch Case Expression, as many as required are then followed by a final column validation expression,
	//used if none of the Switch Case Expressions evaluate to true.